Oct 30 11:53:26.916532 ip-10-0-138-23 systemd[1]: Starting Kubernetes Kubelet... Oct 30 11:53:27.262381 ip-10-0-138-23 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:53:27.262381 ip-10-0-138-23 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 11:53:27.262381 ip-10-0-138-23 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:53:27.262381 ip-10-0-138-23 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:53:27.267132 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267055 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:27.267132 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267133 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:27.267132 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267136 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267142 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267145 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267147 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267150 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267153 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267156 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267159 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267162 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267164 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267167 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267170 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267172 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267175 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267178 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267180 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267183 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267185 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267188 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267198 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:27.267231 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267201 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267203 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267206 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267210 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267215 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267218 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267221 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267223 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267226 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267229 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267231 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267234 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267236 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267241 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267259 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267263 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267265 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267268 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267271 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:27.267733 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267273 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267276 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267278 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267281 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267283 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267286 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267288 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267291 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267293 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267296 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267299 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267301 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267304 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267307 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267309 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267313 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267316 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267319 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267321 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267325 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:27.268183 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267328 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267330 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267333 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267335 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267338 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267340 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267343 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267345 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267348 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267350 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267353 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267356 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267359 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267361 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267364 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267367 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267369 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267371 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267374 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:27.268710 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267376 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267379 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267381 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267384 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267386 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267389 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267391 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267394 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267404 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267407 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267410 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267413 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267416 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267418 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267421 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267423 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267426 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267430 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267432 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267435 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:27.269156 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267437 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267440 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267443 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267445 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267448 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.267451 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268219 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268225 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268228 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268230 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268233 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268236 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268239 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268242 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268261 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268264 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268267 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268270 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268275 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268277 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:27.269645 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268280 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268283 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268286 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268289 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268291 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268294 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268296 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268299 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268301 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268304 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268306 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268309 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268312 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268314 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268317 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268320 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268324 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268327 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268331 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:27.270126 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268334 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268337 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268340 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268342 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268345 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268347 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268350 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268352 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268355 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268357 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268360 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268364 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268366 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268371 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268373 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268376 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268379 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268382 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268384 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:27.270589 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268386 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268391 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268394 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268397 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268399 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268401 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268404 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268407 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268409 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268412 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268414 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268417 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268419 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268425 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268428 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268430 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268432 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268435 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268437 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:27.271050 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268440 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268442 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268445 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268448 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268451 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268462 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268465 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268468 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268470 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268473 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268475 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268478 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268481 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268483 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268486 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268489 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268492 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268494 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268497 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:27.271509 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268499 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268502 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268504 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268507 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268509 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268512 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268515 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268517 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268528 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268532 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268535 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268538 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268541 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268543 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268546 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.268549 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269105 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269112 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269117 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269120 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:27.271959 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269124 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269127 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269130 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269132 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269135 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269137 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269140 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269143 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269145 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269148 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269151 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269153 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269156 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269159 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269162 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269165 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269167 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269170 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269173 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:27.272586 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269176 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269179 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269182 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269184 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269187 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269189 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269192 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269194 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269197 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269199 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269202 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269206 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269209 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269211 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269214 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269217 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269219 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269222 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269225 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:27.273056 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269229 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269232 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269235 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269237 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269240 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269243 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269259 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269262 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269265 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269267 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269270 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269273 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269276 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269279 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269281 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269285 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269287 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269290 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269292 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269295 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:27.273519 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269297 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269300 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269302 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269305 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269308 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269311 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269314 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269316 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269319 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269323 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269325 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269328 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269331 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269333 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269336 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269338 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269341 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269343 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269346 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269348 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:27.274028 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269351 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269354 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269356 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269360 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269363 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269366 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269368 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269371 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269374 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269377 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269380 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269382 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269385 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269387 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269390 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269392 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269396 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269399 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269401 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269404 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:27.274504 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269407 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269410 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269412 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.269415 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270180 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270189 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270195 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270200 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270204 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270207 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270211 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270215 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270218 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270221 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270225 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270230 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270233 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270236 2577 flags.go:64] FLAG: --cgroup-root="" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270239 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270241 2577 flags.go:64] FLAG: --client-ca-file="" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270260 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270267 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270274 2577 flags.go:64] FLAG: --cluster-domain="" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270277 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 11:53:27.274989 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270280 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270284 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270287 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270291 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270294 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270297 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270301 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270304 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270307 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270309 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270312 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270315 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270320 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270323 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270325 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270328 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270331 2577 flags.go:64] FLAG: --enable-server="true" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270334 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270338 2577 flags.go:64] FLAG: --event-burst="100" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270341 2577 flags.go:64] FLAG: --event-qps="50" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270344 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270347 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270350 2577 flags.go:64] FLAG: --eviction-hard="" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270356 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 11:53:27.275621 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270359 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270362 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270365 2577 flags.go:64] FLAG: --eviction-soft="" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270368 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270372 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270375 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270378 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270381 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270384 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270387 2577 flags.go:64] FLAG: --feature-gates="" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270391 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270394 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270397 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270400 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270403 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270406 2577 flags.go:64] FLAG: --help="false" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270409 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-138-23.ec2.internal" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270412 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270415 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270418 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270421 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270424 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270427 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 11:53:27.276265 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270430 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270433 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270435 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270438 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270441 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270444 2577 flags.go:64] FLAG: --kube-reserved="" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270447 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270450 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270453 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270456 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270459 2577 flags.go:64] FLAG: --lock-file="" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270462 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270465 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270468 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270473 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270476 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270480 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270482 2577 flags.go:64] FLAG: --logging-format="text" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270485 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270489 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270491 2577 flags.go:64] FLAG: --manifest-url="" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270494 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270498 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270501 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270505 2577 flags.go:64] FLAG: --max-pods="110" Oct 30 11:53:27.276926 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270508 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270511 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270514 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270516 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270519 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270522 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270525 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270531 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270534 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270537 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270540 2577 flags.go:64] FLAG: --pod-cidr="" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270543 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270546 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270549 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270551 2577 flags.go:64] FLAG: --port="10250" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270555 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270558 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a7d4e5c9786b6093" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270561 2577 flags.go:64] FLAG: --qos-reserved="" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270565 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270568 2577 flags.go:64] FLAG: --register-node="true" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270571 2577 flags.go:64] FLAG: --register-with-taints="" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270574 2577 flags.go:64] FLAG: --registry-burst="10" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270577 2577 flags.go:64] FLAG: --registry-qps="5" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270580 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270584 2577 flags.go:64] FLAG: --reserved-memory="" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270588 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 11:53:27.277794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270591 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270594 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270597 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270600 2577 flags.go:64] FLAG: --runonce="false" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270603 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270606 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270609 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270612 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270615 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270618 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270621 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270624 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270626 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270629 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270632 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270635 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270638 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270641 2577 flags.go:64] FLAG: --system-cgroups="" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270644 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270649 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270652 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270655 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270659 2577 flags.go:64] FLAG: --tls-min-version="" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270662 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270666 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 11:53:27.279064 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270669 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270672 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270674 2577 flags.go:64] FLAG: --v="2" Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270679 2577 flags.go:64] FLAG: --version="false" Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270683 2577 flags.go:64] FLAG: --vmodule="" Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270687 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.270691 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270834 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270837 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270840 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270843 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270846 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270849 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270851 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270854 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270856 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270859 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270861 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270864 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270866 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270869 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:27.279732 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270871 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270874 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270876 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270879 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270883 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270887 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270890 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270893 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270896 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270900 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270904 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270907 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270910 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270912 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270915 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270918 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270920 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270923 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270926 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270928 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:27.280276 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270931 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270933 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270936 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270938 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270941 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270944 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270946 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270948 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270951 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270953 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270956 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270958 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270961 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270964 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270967 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270969 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270972 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270974 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270977 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270980 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:27.280774 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270982 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270985 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270990 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270993 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270995 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.270998 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271000 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271003 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271005 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271008 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271010 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271014 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271016 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271019 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271021 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271024 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271026 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271029 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271031 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:27.281242 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271035 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271039 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271043 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271045 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271048 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271050 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271053 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271056 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271058 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271061 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271064 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271066 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271069 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271071 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271074 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271078 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271080 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271083 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271085 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271088 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:27.281707 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271090 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:27.282163 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271093 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:27.282163 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271096 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:27.282163 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271098 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:27.282163 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271100 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:27.282163 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271103 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:27.282163 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271106 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:27.282163 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271108 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:27.282163 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271112 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:27.282163 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271114 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:27.282163 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271116 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:27.282163 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271119 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:27.282163 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271122 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:27.282163 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.271149 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:53:27.282163 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271195 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:27.282163 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271200 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:27.282163 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271203 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271207 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271210 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271213 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271215 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271218 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271220 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271223 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271227 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271230 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271232 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271235 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271240 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271242 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271261 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271265 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271268 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271271 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271273 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:27.282564 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271276 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271278 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271281 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271284 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271287 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271289 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271292 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271294 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271297 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271299 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271302 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271304 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271307 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271310 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271312 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271315 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271317 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271320 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271322 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271325 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:27.283005 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271327 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271343 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271346 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271349 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271352 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271356 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271359 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271362 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271364 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271367 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271370 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271373 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271375 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271378 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271380 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271383 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271386 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271388 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271391 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:27.283492 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271394 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271396 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271399 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271401 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271404 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271406 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271413 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271418 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271421 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271425 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271427 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271430 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271432 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271435 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271438 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271441 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271444 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271446 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271450 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271453 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:27.283955 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271455 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271458 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271461 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271463 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271466 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271468 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271471 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271474 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271476 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271479 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271482 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271484 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271487 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271489 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271492 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271495 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271497 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271500 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271502 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271505 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:27.284528 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271508 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271510 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271513 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271516 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271518 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.271521 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.279132 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.279146 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.279163 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.279168 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279229 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279233 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279236 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279239 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279242 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279259 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279263 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279265 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279268 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279270 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:27.285000 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279273 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279275 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279278 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279281 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279284 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279288 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279291 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279293 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279296 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279298 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279301 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279303 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279306 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279308 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279311 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279313 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279316 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279319 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279323 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:27.285517 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279325 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279328 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279330 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279333 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279335 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279338 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279340 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279343 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279345 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279349 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279353 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279356 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279359 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279361 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279364 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279366 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279369 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279371 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279374 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:27.285964 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279376 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279378 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279381 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279384 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279386 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279388 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279391 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279394 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279396 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279399 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279402 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279405 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279407 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279409 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279412 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279414 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279417 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279420 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279422 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279425 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:27.286452 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279427 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279429 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279432 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279434 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279437 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279439 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279442 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279444 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279447 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279449 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279452 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279455 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279457 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279460 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279462 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279465 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279467 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279469 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279472 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279474 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:27.286913 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279477 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:27.287468 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279479 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:27.287468 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279482 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:27.287468 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279484 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:27.287468 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279487 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:27.287468 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279489 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:27.287468 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279492 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:27.287468 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279495 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:27.287468 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279497 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:27.287468 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279500 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:27.287468 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279502 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:27.287468 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279505 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:27.287468 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279507 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:27.287468 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279510 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:27.287468 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279512 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:27.287468 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279515 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:27.287468 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279517 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:27.287468 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279520 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:27.287854 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.279546 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:53:27.287854 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279680 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:27.287854 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279685 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:27.287854 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279688 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:27.287854 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279691 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:27.287854 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279693 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:27.287854 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279697 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:27.287854 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279699 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:27.287854 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279702 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:27.287854 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279705 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:27.287854 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279707 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:27.287854 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279710 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:27.287854 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279712 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:27.287854 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279715 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:27.287854 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279717 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:27.287854 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279720 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279723 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279744 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279748 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279751 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279755 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279758 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279761 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279764 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279767 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279769 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279772 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279774 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279777 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279779 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279782 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279784 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279787 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279789 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279791 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:27.288295 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279794 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279797 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279800 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279802 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279805 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279808 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279810 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279813 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279815 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279818 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279820 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279823 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279825 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279828 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279830 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279833 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279835 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279837 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279842 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:27.288829 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279844 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279847 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279849 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279852 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279854 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279857 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279860 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279862 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279864 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279867 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279870 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279872 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279875 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279878 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279881 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279883 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279886 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279889 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279892 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279894 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:27.289289 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279897 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279899 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279902 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279904 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279908 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279911 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279914 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279917 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279920 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279922 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279925 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279928 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279930 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279933 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279936 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279938 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279941 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279943 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279946 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:27.289841 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279949 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:27.290503 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279951 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:27.290503 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279954 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:27.290503 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279956 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:27.290503 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279959 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:27.290503 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279961 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:27.290503 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279965 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:27.290503 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279967 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:27.290503 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279970 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:27.290503 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279972 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:27.290503 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279976 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:27.290503 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279979 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:27.290503 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279983 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:27.290503 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.279985 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:27.290503 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.280008 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:53:27.290503 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280040 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:53:27.290503 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280044 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280047 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280050 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280053 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280056 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280061 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280064 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280067 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280070 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280073 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280076 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280078 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280081 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280083 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280086 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280088 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280091 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280094 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280096 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:53:27.290893 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280099 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280101 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280104 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280108 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280110 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280113 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280115 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280118 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280120 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280123 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280127 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280131 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280134 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280137 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280139 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280142 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280145 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280147 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280150 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280152 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:53:27.291688 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280155 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280157 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280161 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280163 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280166 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280168 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280171 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280173 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280176 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280179 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280182 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280184 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280187 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280189 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280192 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280195 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280198 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280201 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280204 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:53:27.292205 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280206 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280209 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280211 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280213 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280216 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280219 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280222 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280224 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280227 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280229 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280232 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280234 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280237 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280239 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280242 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280262 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280265 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280268 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280270 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280274 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:53:27.292764 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280276 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280279 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280282 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280284 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280287 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280290 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280293 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280295 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280299 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280301 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280304 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280307 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280309 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280312 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280315 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280317 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280320 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280323 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280327 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:53:27.293281 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280329 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:53:27.293727 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280332 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:53:27.293727 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280334 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:53:27.293727 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280337 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:53:27.293727 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280339 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:53:27.293727 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280342 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:53:27.293727 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280344 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:53:27.293727 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:27.280347 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:53:27.293727 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.280395 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 11:53:27.293727 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.283656 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 11:53:27.293727 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.284438 2577 server.go:1014] "Starting client certificate rotation" Oct 30 11:53:27.293727 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.284549 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 11:53:27.293727 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.285180 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 11:53:27.293727 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.286125 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 11:53:27.293727 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.287180 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 11:53:27.296327 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.296279 2577 log.go:25] "Validated CRI v1 runtime API" Oct 30 11:53:27.299977 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.299963 2577 log.go:25] "Validated CRI v1 image API" Oct 30 11:53:27.300856 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.300834 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 11:53:27.303860 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.303841 2577 fs.go:135] Filesystem UUIDs: map[39daed18-86e6-4776-868c-7cd05318025c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f1f48a21-c751-4b0b-870d-d1d7bc18057b:/dev/nvme0n1p3] Oct 30 11:53:27.303956 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.303861 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 11:53:27.310608 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.310486 2577 manager.go:217] Machine: {Timestamp:2026-10-30 11:53:27.309041825 +0000 UTC m=+0.287734615 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3115501 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28c64b7420621ac5361e53ab0550a8 SystemUUID:ec28c64b-7420-621a-c536-1e53ab0550a8 BootID:1ddfa322-0634-48ae-b3e0-a31d7cad26d0 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e9:0a:87:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e9:0a:87:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:4f:11:f1:08:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 11:53:27.310608 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.310596 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 11:53:27.310738 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.310697 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 11:53:27.313244 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.313213 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 11:53:27.313387 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.313262 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-23.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 11:53:27.313428 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.313390 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 11:53:27.313428 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.313399 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 11:53:27.313489 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.313439 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 11:53:27.313489 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.313458 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 11:53:27.314127 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.314118 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 11:53:27.317940 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.317928 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 11:53:27.318078 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.318069 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 11:53:27.318646 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.318634 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 30 11:53:27.318676 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.318652 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 11:53:27.318676 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.318663 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 11:53:27.318676 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.318672 2577 kubelet.go:405] "Adding apiserver pod source" Oct 30 11:53:27.318778 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.318680 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 11:53:27.318981 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.318970 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 11:53:27.320853 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.320836 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 11:53:27.325877 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.325864 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 11:53:27.326740 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.326728 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 11:53:27.326773 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.326759 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 11:53:27.326904 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.326894 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 11:53:27.326936 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.326911 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 11:53:27.326936 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.326918 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 11:53:27.326936 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.326923 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 11:53:27.326936 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.326929 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 11:53:27.326936 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.326935 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 11:53:27.327408 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.327400 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 11:53:27.327446 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.327410 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 11:53:27.327446 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.327417 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 11:53:27.327446 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.327423 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 11:53:27.327446 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.327438 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 11:53:27.327446 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.327447 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 11:53:27.327954 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.327946 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 11:53:27.327986 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.327955 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 11:53:27.332431 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.332415 2577 server.go:1263] "Started kubelet" Oct 30 11:53:27.332575 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.332547 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 11:53:27.332644 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.332598 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 11:53:27.332693 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.332667 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 11:53:27.333742 ip-10-0-138-23 systemd[1]: Started Kubernetes Kubelet. Oct 30 11:53:27.333892 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.333741 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 11:53:27.334750 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.334734 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 30 11:53:27.339502 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.339482 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 11:53:27.339916 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.339900 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 11:53:27.340065 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.340046 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found" Oct 30 11:53:27.340174 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.339519 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 11:53:27.340977 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.340736 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 11:53:27.340977 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.340769 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 11:53:27.340977 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.340876 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 11:53:27.341159 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.341033 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 30 11:53:27.341159 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.341042 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 30 11:53:27.341271 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.341186 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 11:53:27.341322 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.341274 2577 factory.go:55] Registering systemd factory Oct 30 11:53:27.341322 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.341290 2577 factory.go:223] Registration of the systemd container factory successfully Oct 30 11:53:27.343194 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.343177 2577 factory.go:153] Registering CRI-O factory Oct 30 11:53:27.343194 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.343193 2577 factory.go:223] Registration of the crio container factory successfully Oct 30 11:53:27.343324 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.343211 2577 factory.go:103] Registering Raw factory Oct 30 11:53:27.343324 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.343222 2577 manager.go:1199] Started watching for new ooms in manager Oct 30 11:53:27.344029 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.344014 2577 manager.go:319] Starting recovery of all containers Oct 30 11:53:27.353202 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.353186 2577 manager.go:324] Recovery completed Oct 30 11:53:27.356902 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.356890 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:27.358885 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.358862 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:27.358965 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.358900 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:27.358965 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.358911 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:27.359499 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.359485 2577 cpu_manager.go:226] "Starting" policy="none" Oct 30 11:53:27.359499 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.359499 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 11:53:27.359626 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.359515 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 11:53:27.359626 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.359489 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:27.360166 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.360147 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:27.360228 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.360175 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:27.360228 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.360185 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:27.361688 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.361677 2577 policy_none.go:50] "Start" Oct 30 11:53:27.361728 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.361691 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 11:53:27.361728 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.361703 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 11:53:27.363579 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.363568 2577 policy_none.go:44] "Start" Oct 30 11:53:27.368363 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.368289 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-23.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 11:53:27.368363 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.368293 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-23.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 11:53:27.369184 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.368310 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-23.ec2.internal.18e34c8435030941 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-23.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-23.ec2.internal,},FirstTimestamp:2026-10-30 11:53:27.332374849 +0000 UTC m=+0.311067637,LastTimestamp:2026-10-30 11:53:27.332374849 +0000 UTC m=+0.311067637,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-23.ec2.internal,}" Oct 30 11:53:27.382198 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.382119 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-23.ec2.internal.18e34c8436979364 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-23.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-23.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-23.ec2.internal,},FirstTimestamp:2026-10-30 11:53:27.358886756 +0000 UTC m=+0.337579555,LastTimestamp:2026-10-30 11:53:27.358886756 +0000 UTC m=+0.337579555,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-23.ec2.internal,}" Oct 30 11:53:27.382360 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.382227 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.395493 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.395546 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6j9pc" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.402410 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.402446 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-23.ec2.internal.18e34c843697dd29 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-23.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-23.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-23.ec2.internal,},FirstTimestamp:2026-10-30 11:53:27.358905641 +0000 UTC m=+0.337598431,LastTimestamp:2026-10-30 11:53:27.358905641 +0000 UTC m=+0.337598431,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-23.ec2.internal,}" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.402541 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-23.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.402621 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6j9pc" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.405810 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.407099 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.407121 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.407140 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.407177 2577 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.410444 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.413105 2577 manager.go:341] "Starting Device Plugin manager" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.413128 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.413136 2577 server.go:87] "Starting device plugin registration server" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.413346 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.413355 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.413447 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.413522 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.413530 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.413965 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.414006 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-23.ec2.internal\" not found" Oct 30 11:53:27.418707 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.418550 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-23.ec2.internal.18e34c843698023d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-23.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-23.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-23.ec2.internal,},FirstTimestamp:2026-10-30 11:53:27.358915133 +0000 UTC m=+0.337607925,LastTimestamp:2026-10-30 11:53:27.358915133 +0000 UTC m=+0.337607925,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-23.ec2.internal,}" Oct 30 11:53:27.443262 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.443229 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:53:27.507900 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.507873 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal"] Oct 30 11:53:27.507979 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.507941 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:27.510189 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.510176 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:27.510268 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.510201 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:27.510268 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.510210 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:27.512337 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.512325 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:27.512520 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.512469 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:53:27.512520 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.512499 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:27.513042 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.513029 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:27.513097 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.513054 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:27.513097 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.513063 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:27.513160 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.513028 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:27.513160 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.513130 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:27.513160 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.513143 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:27.513714 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.513702 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:27.514387 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.514369 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:27.514443 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.514402 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:27.514443 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.514416 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:27.514443 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.514440 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-23.ec2.internal" Oct 30 11:53:27.515807 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.515787 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" Oct 30 11:53:27.515885 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.515811 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:53:27.516514 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.516494 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:53:27.516600 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.516526 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:53:27.516600 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.516539 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:53:27.532567 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.532552 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-23.ec2.internal\" not found" node="ip-10-0-138-23.ec2.internal" Oct 30 11:53:27.534149 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.534136 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-23.ec2.internal" Oct 30 11:53:27.534191 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.534155 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-23.ec2.internal\": node \"ip-10-0-138-23.ec2.internal\" not found" Oct 30 11:53:27.534419 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.534402 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-23.ec2.internal\" not found" node="ip-10-0-138-23.ec2.internal" Oct 30 11:53:27.586475 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.586460 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found" Oct 30 11:53:27.641842 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.641819 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e9967fa0ac01d9b4f8c399a65ef8efb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" (UID: \"3e9967fa0ac01d9b4f8c399a65ef8efb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:53:27.641949 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.641844 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e9967fa0ac01d9b4f8c399a65ef8efb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" (UID: \"3e9967fa0ac01d9b4f8c399a65ef8efb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:53:27.641949 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.641860 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8314998578de81aba897c7d170293e92-config\") pod \"kube-apiserver-proxy-ip-10-0-138-23.ec2.internal\" (UID: \"8314998578de81aba897c7d170293e92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" Oct 30 11:53:27.687187 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.687170 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found" Oct 30 11:53:27.742074 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.742051 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8314998578de81aba897c7d170293e92-config\") pod \"kube-apiserver-proxy-ip-10-0-138-23.ec2.internal\" (UID: \"8314998578de81aba897c7d170293e92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" Oct 30 11:53:27.742159 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.742085 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e9967fa0ac01d9b4f8c399a65ef8efb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" (UID: \"3e9967fa0ac01d9b4f8c399a65ef8efb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:53:27.742159 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.742109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e9967fa0ac01d9b4f8c399a65ef8efb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" (UID: \"3e9967fa0ac01d9b4f8c399a65ef8efb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:53:27.742322 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.742166 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8314998578de81aba897c7d170293e92-config\") pod \"kube-apiserver-proxy-ip-10-0-138-23.ec2.internal\" (UID: \"8314998578de81aba897c7d170293e92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" Oct 30 11:53:27.742322 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.742164 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e9967fa0ac01d9b4f8c399a65ef8efb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" (UID: \"3e9967fa0ac01d9b4f8c399a65ef8efb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:53:27.742322 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.742167 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e9967fa0ac01d9b4f8c399a65ef8efb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" (UID: \"3e9967fa0ac01d9b4f8c399a65ef8efb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:53:27.787726 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.787672 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found" Oct 30 11:53:27.843569 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.843542 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:53:27.843569 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:27.843563 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" Oct 30 11:53:27.888668 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.888634 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found" Oct 30 11:53:27.988852 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:27.988827 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found" Oct 30 11:53:28.089334 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:28.089266 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found" Oct 30 11:53:28.189824 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:28.189790 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found" Oct 30 11:53:28.284315 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:28.284288 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 11:53:28.284772 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:28.284471 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 11:53:28.290488 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:28.290471 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found" Oct 30 11:53:28.365858 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:28.365675 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9967fa0ac01d9b4f8c399a65ef8efb.slice/crio-17b883f17b8183ee1f8d45b16f43ee7d69bd5f0cf23fac2fbf8fff150588df71 WatchSource:0}: Error finding container 17b883f17b8183ee1f8d45b16f43ee7d69bd5f0cf23fac2fbf8fff150588df71: Status 404 returned error can't find the container with id 17b883f17b8183ee1f8d45b16f43ee7d69bd5f0cf23fac2fbf8fff150588df71 Oct 30 11:53:28.366095 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:28.366077 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8314998578de81aba897c7d170293e92.slice/crio-376ea76ae9c666fdffe54b57c639f33b6068a1b0f6df2c719aa300065016161e WatchSource:0}: Error finding container 376ea76ae9c666fdffe54b57c639f33b6068a1b0f6df2c719aa300065016161e: Status 404 returned error can't find the container with id 376ea76ae9c666fdffe54b57c639f33b6068a1b0f6df2c719aa300065016161e Oct 30 11:53:28.370701 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:28.370685 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 11:53:28.390575 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:28.390554 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found" Oct 30 11:53:28.407367 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:28.407336 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 11:48:28 +0000 UTC" deadline="2028-08-16 12:55:29.632493486 +0000 UTC" Oct 30 11:53:28.407367 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:28.407360 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15745h2m1.225137722s" Oct 30 11:53:28.410378 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:28.410334 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" event={"ID":"8314998578de81aba897c7d170293e92","Type":"ContainerStarted","Data":"376ea76ae9c666fdffe54b57c639f33b6068a1b0f6df2c719aa300065016161e"} Oct 30 11:53:28.411295 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:28.411277 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" event={"ID":"3e9967fa0ac01d9b4f8c399a65ef8efb","Type":"ContainerStarted","Data":"17b883f17b8183ee1f8d45b16f43ee7d69bd5f0cf23fac2fbf8fff150588df71"} Oct 30 11:53:28.491618 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:28.491597 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found" Oct 30 11:53:28.508542 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:28.508519 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:53:28.540520 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:28.540502 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:53:28.567670 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:28.567655 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:53:28.569200 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:28.569188 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" Oct 30 11:53:28.589575 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:28.589557 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:53:28.590022 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:28.590008 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:53:28.684203 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:28.684139 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:53:29.320171 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.320144 2577 apiserver.go:52] "Watching apiserver" Oct 30 11:53:29.340169 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.340131 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 11:53:29.350368 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.350346 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 11:53:29.350790 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.350761 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-7lb7d","openshift-image-registry/node-ca-f64ck","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6","openshift-multus/multus-gshq6","openshift-multus/network-metrics-daemon-qvs98","openshift-network-diagnostics/network-check-target-g584t","openshift-network-operator/iptables-alerter-s96m4","openshift-dns/node-resolver-jz62m","openshift-multus/multus-additional-cni-plugins-hchnk","kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal","openshift-cluster-node-tuning-operator/tuned-dmj7l","openshift-ovn-kubernetes/ovnkube-node-kwcqk","kube-system/global-pull-secret-syncer-pnvz9"] Oct 30 11:53:29.353668 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.353640 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:29.353767 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:29.353725 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g584t" podUID="8b4fd8cb-a92d-40b2-8133-b04c126332e7" Oct 30 11:53:29.356082 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.356055 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.358508 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.358484 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:29.358666 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:29.358568 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnvz9" podUID="68aaec5d-0bc6-4e8b-a311-1797a055bb7b" Oct 30 11:53:29.361020 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.361002 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.363443 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.363420 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.365845 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.365813 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:29.365958 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:29.365900 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvs98" podUID="4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd" Oct 30 11:53:29.368361 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.368339 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jz62m" Oct 30 11:53:29.372603 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.372582 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7lb7d" Oct 30 11:53:29.374033 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.374009 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8n4dr\"" Oct 30 11:53:29.374148 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.374116 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 11:53:29.375214 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.375194 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 11:53:29.375397 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.375371 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 11:53:29.375488 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.375397 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 11:53:29.375605 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.375576 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 11:53:29.375705 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.375629 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 11:53:29.377034 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.376838 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 11:53:29.377034 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.376895 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 11:53:29.377034 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.376952 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f64ck" Oct 30 11:53:29.378713 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.378603 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 11:53:29.378713 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.378680 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6hs8x\"" Oct 30 11:53:29.378878 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.378790 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.381231 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.381211 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s96m4" Oct 30 11:53:29.383791 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.383770 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.387814 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.387770 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 11:53:29.387814 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.387772 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 11:53:29.387970 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.387838 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-llkx2\"" Oct 30 11:53:29.387970 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.387874 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 11:53:29.389184 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.389135 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 11:53:29.389184 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.389155 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 11:53:29.389362 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.389198 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 11:53:29.389362 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.389248 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 11:53:29.389362 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.389268 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 11:53:29.389362 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.389299 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7cm7r\"" Oct 30 11:53:29.389362 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.389337 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lrxbh\"" Oct 30 11:53:29.390187 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.390146 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 11:53:29.396599 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.395578 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wszhh" Oct 30 11:53:29.410043 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.410020 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wszhh" Oct 30 11:53:29.441938 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.441918 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 11:53:29.451824 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.451804 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-node-log\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.451944 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.451833 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-var-lib-cni-bin\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.451944 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.451860 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.451944 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.451885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edbd4ad4-8205-460e-9385-3151a16e7b68-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.451944 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.451905 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc833d39-b781-486b-9fcc-549cba555ed9-etc-tuned\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.451944 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.451924 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-etc-openvswitch\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.452191 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.451950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-run-ovn\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.452191 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.451974 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbv7k\" (UniqueName: \"kubernetes.io/projected/52ede1dc-817e-434c-874a-2db2a9c10f52-kube-api-access-kbv7k\") pod \"iptables-alerter-s96m4\" (UID: \"52ede1dc-817e-434c-874a-2db2a9c10f52\") " pod="openshift-network-operator/iptables-alerter-s96m4" Oct 30 11:53:29.452191 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.451999 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-run-netns\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.452191 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452014 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-cni-netd\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.452191 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452030 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c3ac2bd-4472-4779-8ca5-460160cdedbb-ovn-node-metrics-cert\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.452191 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452044 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a3afa91-aff6-455b-aeda-7275cd8ac8e0-agent-certs\") pod \"konnectivity-agent-7lb7d\" (UID: \"1a3afa91-aff6-455b-aeda-7275cd8ac8e0\") " pod="kube-system/konnectivity-agent-7lb7d" Oct 30 11:53:29.452191 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452060 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-sys\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.452191 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452082 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-etc-kubernetes\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.452191 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452097 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52ede1dc-817e-434c-874a-2db2a9c10f52-host-slash\") pod \"iptables-alerter-s96m4\" (UID: \"52ede1dc-817e-434c-874a-2db2a9c10f52\") " pod="openshift-network-operator/iptables-alerter-s96m4" Oct 30 11:53:29.452191 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452112 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs\") pod \"network-metrics-daemon-qvs98\" (UID: \"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd\") " pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:29.452191 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452135 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzcmw\" (UniqueName: \"kubernetes.io/projected/fc833d39-b781-486b-9fcc-549cba555ed9-kube-api-access-hzcmw\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.452191 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452160 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27f694f1-e1de-483f-a5c5-250f52dae768-host\") pod \"node-ca-f64ck\" (UID: \"27f694f1-e1de-483f-a5c5-250f52dae768\") " pod="openshift-image-registry/node-ca-f64ck" Oct 30 11:53:29.452191 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452186 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nqkz\" (UniqueName: \"kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz\") pod \"network-check-target-g584t\" (UID: \"8b4fd8cb-a92d-40b2-8133-b04c126332e7\") " pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:29.452794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452211 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c3ac2bd-4472-4779-8ca5-460160cdedbb-ovnkube-config\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.452794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452232 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c3ac2bd-4472-4779-8ca5-460160cdedbb-ovnkube-script-lib\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.452794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452267 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edbd4ad4-8205-460e-9385-3151a16e7b68-system-cni-dir\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.452794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452298 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-kubernetes\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.452794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452348 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-multus-cni-dir\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.452794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452379 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-var-lib-openvswitch\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.452794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452404 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-dbus\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:29.452794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452438 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-registration-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.452794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-sys-fs\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.452794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-var-lib-kubelet\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.452794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452528 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/52ede1dc-817e-434c-874a-2db2a9c10f52-iptables-alerter-script\") pod \"iptables-alerter-s96m4\" (UID: \"52ede1dc-817e-434c-874a-2db2a9c10f52\") " pod="openshift-network-operator/iptables-alerter-s96m4" Oct 30 11:53:29.452794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452546 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.452794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452561 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brmjx\" (UniqueName: \"kubernetes.io/projected/1c3ac2bd-4472-4779-8ca5-460160cdedbb-kube-api-access-brmjx\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.452794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:29.452794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452599 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edbd4ad4-8205-460e-9385-3151a16e7b68-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.452794 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452627 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-cnibin\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.453579 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452647 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/864715b0-07ec-483a-8401-089c747422cb-multus-daemon-config\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.453579 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452662 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27f694f1-e1de-483f-a5c5-250f52dae768-serviceca\") pod \"node-ca-f64ck\" (UID: \"27f694f1-e1de-483f-a5c5-250f52dae768\") " pod="openshift-image-registry/node-ca-f64ck" Oct 30 11:53:29.453579 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452680 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-systemd-units\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.453579 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx4gc\" (UniqueName: \"kubernetes.io/projected/e2b6ec57-b883-408a-ab01-abd192572510-kube-api-access-vx4gc\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.453579 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452711 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhm5c\" (UniqueName: \"kubernetes.io/projected/edbd4ad4-8205-460e-9385-3151a16e7b68-kube-api-access-hhm5c\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.453579 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452737 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-systemd\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.453579 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452762 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-lib-modules\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.453579 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc833d39-b781-486b-9fcc-549cba555ed9-tmp\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.453579 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452813 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-multus-socket-dir-parent\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.453579 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452838 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-run-multus-certs\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.453579 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452857 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-kubelet\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.453579 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452873 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-run-systemd\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.453579 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452888 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-log-socket\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.453579 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452914 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-cni-bin\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.453579 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452939 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-kubelet-config\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:29.453579 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452956 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-socket-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.454152 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452973 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edbd4ad4-8205-460e-9385-3151a16e7b68-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.454152 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.452989 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7dwl\" (UniqueName: \"kubernetes.io/projected/864715b0-07ec-483a-8401-089c747422cb-kube-api-access-w7dwl\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.454152 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453012 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-slash\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.454152 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453038 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c3ac2bd-4472-4779-8ca5-460160cdedbb-env-overrides\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.454152 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453056 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-etc-selinux\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.454152 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453072 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a3afa91-aff6-455b-aeda-7275cd8ac8e0-konnectivity-ca\") pod \"konnectivity-agent-7lb7d\" (UID: \"1a3afa91-aff6-455b-aeda-7275cd8ac8e0\") " pod="kube-system/konnectivity-agent-7lb7d" Oct 30 11:53:29.454152 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-sysconfig\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.454152 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453124 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-system-cni-dir\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.454152 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453161 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/864715b0-07ec-483a-8401-089c747422cb-cni-binary-copy\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.454152 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453188 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-run-k8s-cni-cncf-io\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.454152 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453212 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edbd4ad4-8205-460e-9385-3151a16e7b68-cni-binary-copy\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.454152 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453235 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-run\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.454152 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453261 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-multus-conf-dir\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.454152 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453288 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/535eda1f-d9e7-4908-b958-56f531b68fbb-hosts-file\") pod \"node-resolver-jz62m\" (UID: \"535eda1f-d9e7-4908-b958-56f531b68fbb\") " pod="openshift-dns/node-resolver-jz62m" Oct 30 11:53:29.454152 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/535eda1f-d9e7-4908-b958-56f531b68fbb-tmp-dir\") pod \"node-resolver-jz62m\" (UID: \"535eda1f-d9e7-4908-b958-56f531b68fbb\") " pod="openshift-dns/node-resolver-jz62m" Oct 30 11:53:29.454152 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453358 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.454893 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-device-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.454893 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453398 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz6mk\" (UniqueName: \"kubernetes.io/projected/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-kube-api-access-tz6mk\") pod \"network-metrics-daemon-qvs98\" (UID: \"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd\") " pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:29.454893 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453421 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edbd4ad4-8205-460e-9385-3151a16e7b68-cnibin\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.454893 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-var-lib-kubelet\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.454893 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453475 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-host\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.454893 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453500 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vkxp\" (UniqueName: \"kubernetes.io/projected/535eda1f-d9e7-4908-b958-56f531b68fbb-kube-api-access-2vkxp\") pod \"node-resolver-jz62m\" (UID: \"535eda1f-d9e7-4908-b958-56f531b68fbb\") " pod="openshift-dns/node-resolver-jz62m" Oct 30 11:53:29.454893 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453538 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-run-openvswitch\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.454893 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453566 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-modprobe-d\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.454893 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453592 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-sysctl-d\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.454893 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453618 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-var-lib-cni-multus\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.454893 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453653 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw9h2\" (UniqueName: \"kubernetes.io/projected/27f694f1-e1de-483f-a5c5-250f52dae768-kube-api-access-bw9h2\") pod \"node-ca-f64ck\" (UID: \"27f694f1-e1de-483f-a5c5-250f52dae768\") " pod="openshift-image-registry/node-ca-f64ck" Oct 30 11:53:29.454893 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453673 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-sysctl-conf\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.454893 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453699 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-run-netns\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.454893 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.453725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-hostroot\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.483793 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.483767 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sw2hc\"" Oct 30 11:53:29.484266 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.484246 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 11:53:29.484425 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.484359 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 11:53:29.484522 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.484509 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 11:53:29.484589 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.484566 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:53:29.484663 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.484616 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:53:29.484663 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.484568 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxkrq\"" Oct 30 11:53:29.484663 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.484574 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 11:53:29.484663 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.484616 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 11:53:29.484663 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.484652 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 11:53:29.484925 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.484676 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 11:53:29.484925 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.484725 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pxw6w\"" Oct 30 11:53:29.484925 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.484742 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tz47t\"" Oct 30 11:53:29.484925 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.484828 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 11:53:29.554443 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554413 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-cnibin\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.554601 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/864715b0-07ec-483a-8401-089c747422cb-multus-daemon-config\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.554601 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554480 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27f694f1-e1de-483f-a5c5-250f52dae768-serviceca\") pod \"node-ca-f64ck\" (UID: \"27f694f1-e1de-483f-a5c5-250f52dae768\") " pod="openshift-image-registry/node-ca-f64ck" Oct 30 11:53:29.554601 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-systemd-units\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.554601 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554540 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx4gc\" (UniqueName: \"kubernetes.io/projected/e2b6ec57-b883-408a-ab01-abd192572510-kube-api-access-vx4gc\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.554601 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554558 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-cnibin\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.554601 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554568 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhm5c\" (UniqueName: \"kubernetes.io/projected/edbd4ad4-8205-460e-9385-3151a16e7b68-kube-api-access-hhm5c\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.554601 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554593 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-systemd\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.554931 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554618 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-lib-modules\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.554931 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc833d39-b781-486b-9fcc-549cba555ed9-tmp\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.554931 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-multus-socket-dir-parent\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.554931 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554699 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-run-multus-certs\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.554931 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-kubelet\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.554931 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554748 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-run-systemd\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.554931 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-log-socket\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.554931 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554799 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-cni-bin\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.554931 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-kubelet-config\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:29.554931 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-socket-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.554931 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edbd4ad4-8205-460e-9385-3151a16e7b68-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.554931 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554908 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-systemd-units\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.554931 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7dwl\" (UniqueName: \"kubernetes.io/projected/864715b0-07ec-483a-8401-089c747422cb-kube-api-access-w7dwl\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.555301 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554943 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-slash\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.555301 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c3ac2bd-4472-4779-8ca5-460160cdedbb-env-overrides\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.555301 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.554966 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-kubelet\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.555301 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555000 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-etc-selinux\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.555301 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a3afa91-aff6-455b-aeda-7275cd8ac8e0-konnectivity-ca\") pod \"konnectivity-agent-7lb7d\" (UID: \"1a3afa91-aff6-455b-aeda-7275cd8ac8e0\") " pod="kube-system/konnectivity-agent-7lb7d" Oct 30 11:53:29.555301 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555051 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-sysconfig\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.555301 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555064 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27f694f1-e1de-483f-a5c5-250f52dae768-serviceca\") pod \"node-ca-f64ck\" (UID: \"27f694f1-e1de-483f-a5c5-250f52dae768\") " pod="openshift-image-registry/node-ca-f64ck" Oct 30 11:53:29.555301 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-system-cni-dir\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.555301 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/864715b0-07ec-483a-8401-089c747422cb-cni-binary-copy\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.555301 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-run-k8s-cni-cncf-io\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.555301 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555149 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edbd4ad4-8205-460e-9385-3151a16e7b68-cni-binary-copy\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.555301 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-run\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.555301 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555181 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/864715b0-07ec-483a-8401-089c747422cb-multus-daemon-config\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.555301 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555198 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-multus-conf-dir\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.555301 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555224 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/535eda1f-d9e7-4908-b958-56f531b68fbb-hosts-file\") pod \"node-resolver-jz62m\" (UID: \"535eda1f-d9e7-4908-b958-56f531b68fbb\") " pod="openshift-dns/node-resolver-jz62m" Oct 30 11:53:29.555301 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555251 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/535eda1f-d9e7-4908-b958-56f531b68fbb-tmp-dir\") pod \"node-resolver-jz62m\" (UID: \"535eda1f-d9e7-4908-b958-56f531b68fbb\") " pod="openshift-dns/node-resolver-jz62m" Oct 30 11:53:29.555301 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555256 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-run-multus-certs\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.555301 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555274 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.555853 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555298 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-device-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.555853 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555325 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz6mk\" (UniqueName: \"kubernetes.io/projected/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-kube-api-access-tz6mk\") pod \"network-metrics-daemon-qvs98\" (UID: \"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd\") " pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:29.555853 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555363 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edbd4ad4-8205-460e-9385-3151a16e7b68-cnibin\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.555853 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555364 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-multus-socket-dir-parent\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.555853 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555363 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-etc-selinux\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.555853 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555405 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-run-systemd\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.555853 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555419 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-run\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.555853 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555425 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-multus-conf-dir\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.555853 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-var-lib-kubelet\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.555853 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555470 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-host\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.555853 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555473 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/535eda1f-d9e7-4908-b958-56f531b68fbb-hosts-file\") pod \"node-resolver-jz62m\" (UID: \"535eda1f-d9e7-4908-b958-56f531b68fbb\") " pod="openshift-dns/node-resolver-jz62m" Oct 30 11:53:29.555853 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555499 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vkxp\" (UniqueName: \"kubernetes.io/projected/535eda1f-d9e7-4908-b958-56f531b68fbb-kube-api-access-2vkxp\") pod \"node-resolver-jz62m\" (UID: \"535eda1f-d9e7-4908-b958-56f531b68fbb\") " pod="openshift-dns/node-resolver-jz62m" Oct 30 11:53:29.555853 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-run-openvswitch\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.555853 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-modprobe-d\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.555853 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555589 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-sysctl-d\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.555853 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-var-lib-cni-multus\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.555853 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw9h2\" (UniqueName: \"kubernetes.io/projected/27f694f1-e1de-483f-a5c5-250f52dae768-kube-api-access-bw9h2\") pod \"node-ca-f64ck\" (UID: \"27f694f1-e1de-483f-a5c5-250f52dae768\") " pod="openshift-image-registry/node-ca-f64ck" Oct 30 11:53:29.556598 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555671 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-sysctl-conf\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.556598 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-run-netns\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.556598 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555719 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 11:53:29.556598 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-hostroot\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.556598 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555765 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/535eda1f-d9e7-4908-b958-56f531b68fbb-tmp-dir\") pod \"node-resolver-jz62m\" (UID: \"535eda1f-d9e7-4908-b958-56f531b68fbb\") " pod="openshift-dns/node-resolver-jz62m" Oct 30 11:53:29.556598 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555774 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-node-log\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.556598 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555804 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-var-lib-cni-bin\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.556598 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555831 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.556598 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555852 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.556598 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edbd4ad4-8205-460e-9385-3151a16e7b68-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.556598 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555887 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc833d39-b781-486b-9fcc-549cba555ed9-etc-tuned\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.556598 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555914 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-etc-openvswitch\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.556598 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-run-ovn\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.556598 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555971 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a3afa91-aff6-455b-aeda-7275cd8ac8e0-konnectivity-ca\") pod \"konnectivity-agent-7lb7d\" (UID: \"1a3afa91-aff6-455b-aeda-7275cd8ac8e0\") " pod="kube-system/konnectivity-agent-7lb7d" Oct 30 11:53:29.556598 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.555975 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbv7k\" (UniqueName: \"kubernetes.io/projected/52ede1dc-817e-434c-874a-2db2a9c10f52-kube-api-access-kbv7k\") pod \"iptables-alerter-s96m4\" (UID: \"52ede1dc-817e-434c-874a-2db2a9c10f52\") " pod="openshift-network-operator/iptables-alerter-s96m4" Oct 30 11:53:29.556598 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-run-netns\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.556598 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-cni-netd\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.556598 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556040 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edbd4ad4-8205-460e-9385-3151a16e7b68-cnibin\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.557371 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556057 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c3ac2bd-4472-4779-8ca5-460160cdedbb-ovn-node-metrics-cert\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.557371 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556092 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-systemd\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.557371 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556107 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-sysconfig\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.557371 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556100 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a3afa91-aff6-455b-aeda-7275cd8ac8e0-agent-certs\") pod \"konnectivity-agent-7lb7d\" (UID: \"1a3afa91-aff6-455b-aeda-7275cd8ac8e0\") " pod="kube-system/konnectivity-agent-7lb7d" Oct 30 11:53:29.557371 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556096 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/864715b0-07ec-483a-8401-089c747422cb-cni-binary-copy\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.557371 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556144 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-log-socket\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.557371 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556148 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-sys\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.557371 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556183 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-system-cni-dir\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.557371 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556189 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-sys\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.557371 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556234 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-lib-modules\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.557371 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556278 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-node-log\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.557371 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556187 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-device-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.557371 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556243 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-run-netns\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.557371 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556312 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-cni-netd\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.557371 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556348 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-var-lib-cni-bin\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.557371 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556367 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-run-netns\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.557371 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556431 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-kubelet-config\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:29.557371 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556495 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-etc-openvswitch\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.558351 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556518 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.558351 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556558 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-run-ovn\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.558351 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556607 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-slash\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.558351 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556028 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-run-k8s-cni-cncf-io\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.558351 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556675 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-etc-kubernetes\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.558351 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52ede1dc-817e-434c-874a-2db2a9c10f52-host-slash\") pod \"iptables-alerter-s96m4\" (UID: \"52ede1dc-817e-434c-874a-2db2a9c10f52\") " pod="openshift-network-operator/iptables-alerter-s96m4" Oct 30 11:53:29.558351 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556734 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-run-openvswitch\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.558351 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556739 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs\") pod \"network-metrics-daemon-qvs98\" (UID: \"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd\") " pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:29.558351 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556785 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzcmw\" (UniqueName: \"kubernetes.io/projected/fc833d39-b781-486b-9fcc-549cba555ed9-kube-api-access-hzcmw\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.558351 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:29.556810 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:29.558351 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:29.556874 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs podName:4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd nodeName:}" failed. No retries permitted until 2026-10-30 11:53:30.056852813 +0000 UTC m=+3.035545610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs") pod "network-metrics-daemon-qvs98" (UID: "4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:29.558351 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556877 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27f694f1-e1de-483f-a5c5-250f52dae768-host\") pod \"node-ca-f64ck\" (UID: \"27f694f1-e1de-483f-a5c5-250f52dae768\") " pod="openshift-image-registry/node-ca-f64ck" Oct 30 11:53:29.558351 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556848 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-cni-bin\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.558351 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556933 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-host\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.558351 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556959 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-var-lib-kubelet\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.558351 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557026 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-etc-kubernetes\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.558351 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557053 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52ede1dc-817e-434c-874a-2db2a9c10f52-host-slash\") pod \"iptables-alerter-s96m4\" (UID: \"52ede1dc-817e-434c-874a-2db2a9c10f52\") " pod="openshift-network-operator/iptables-alerter-s96m4" Oct 30 11:53:29.558351 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.556811 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27f694f1-e1de-483f-a5c5-250f52dae768-host\") pod \"node-ca-f64ck\" (UID: \"27f694f1-e1de-483f-a5c5-250f52dae768\") " pod="openshift-image-registry/node-ca-f64ck" Oct 30 11:53:29.559256 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557103 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nqkz\" (UniqueName: \"kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz\") pod \"network-check-target-g584t\" (UID: \"8b4fd8cb-a92d-40b2-8133-b04c126332e7\") " pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:29.559256 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557133 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c3ac2bd-4472-4779-8ca5-460160cdedbb-ovnkube-config\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.559256 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557145 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-modprobe-d\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.559256 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557161 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c3ac2bd-4472-4779-8ca5-460160cdedbb-ovnkube-script-lib\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.559256 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557188 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edbd4ad4-8205-460e-9385-3151a16e7b68-system-cni-dir\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.559256 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557210 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-var-lib-cni-multus\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.559256 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557215 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-kubernetes\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.559256 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557227 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edbd4ad4-8205-460e-9385-3151a16e7b68-cni-binary-copy\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.559256 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557232 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-sysctl-d\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.559256 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557100 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-hostroot\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.559256 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557285 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-multus-cni-dir\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.559256 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557319 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-var-lib-openvswitch\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.559256 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557365 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-dbus\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:29.559256 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557395 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-registration-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.559256 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557455 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-sys-fs\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.559256 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-var-lib-kubelet\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.559256 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557507 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/52ede1dc-817e-434c-874a-2db2a9c10f52-iptables-alerter-script\") pod \"iptables-alerter-s96m4\" (UID: \"52ede1dc-817e-434c-874a-2db2a9c10f52\") " pod="openshift-network-operator/iptables-alerter-s96m4" Oct 30 11:53:29.560133 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.560133 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brmjx\" (UniqueName: \"kubernetes.io/projected/1c3ac2bd-4472-4779-8ca5-460160cdedbb-kube-api-access-brmjx\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.560133 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:29.560133 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557659 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edbd4ad4-8205-460e-9385-3151a16e7b68-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.560133 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557897 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edbd4ad4-8205-460e-9385-3151a16e7b68-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.560133 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557902 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c3ac2bd-4472-4779-8ca5-460160cdedbb-ovnkube-config\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.560133 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557261 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-kubernetes\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.560133 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557928 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c3ac2bd-4472-4779-8ca5-460160cdedbb-ovnkube-script-lib\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.560133 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557944 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-var-lib-openvswitch\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.560133 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557994 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-socket-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.560133 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.557998 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-dbus\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:29.560133 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.558024 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-registration-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.560133 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.558066 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-sys-fs\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.560133 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.558067 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-multus-cni-dir\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.560133 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.558111 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-var-lib-kubelet\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.560133 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.558110 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c3ac2bd-4472-4779-8ca5-460160cdedbb-env-overrides\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.560133 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.558151 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edbd4ad4-8205-460e-9385-3151a16e7b68-system-cni-dir\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.560133 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:29.558495 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:29.560984 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.558519 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.560984 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:29.558569 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret podName:68aaec5d-0bc6-4e8b-a311-1797a055bb7b nodeName:}" failed. No retries permitted until 2026-10-30 11:53:30.058550966 +0000 UTC m=+3.037243745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret") pod "global-pull-secret-syncer-pnvz9" (UID: "68aaec5d-0bc6-4e8b-a311-1797a055bb7b") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:29.560984 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.558574 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edbd4ad4-8205-460e-9385-3151a16e7b68-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.560984 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.558655 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-sysctl-conf\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.560984 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.558795 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/52ede1dc-817e-434c-874a-2db2a9c10f52-iptables-alerter-script\") pod \"iptables-alerter-s96m4\" (UID: \"52ede1dc-817e-434c-874a-2db2a9c10f52\") " pod="openshift-network-operator/iptables-alerter-s96m4" Oct 30 11:53:29.560984 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.559006 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edbd4ad4-8205-460e-9385-3151a16e7b68-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.561225 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.561052 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc833d39-b781-486b-9fcc-549cba555ed9-tmp\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.561389 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.561263 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc833d39-b781-486b-9fcc-549cba555ed9-etc-tuned\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.562370 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.562347 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a3afa91-aff6-455b-aeda-7275cd8ac8e0-agent-certs\") pod \"konnectivity-agent-7lb7d\" (UID: \"1a3afa91-aff6-455b-aeda-7275cd8ac8e0\") " pod="kube-system/konnectivity-agent-7lb7d" Oct 30 11:53:29.569295 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.569256 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c3ac2bd-4472-4779-8ca5-460160cdedbb-ovn-node-metrics-cert\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.580756 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.580696 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx4gc\" (UniqueName: \"kubernetes.io/projected/e2b6ec57-b883-408a-ab01-abd192572510-kube-api-access-vx4gc\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.580865 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.580384 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhm5c\" (UniqueName: \"kubernetes.io/projected/edbd4ad4-8205-460e-9385-3151a16e7b68-kube-api-access-hhm5c\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.584847 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.584781 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz6mk\" (UniqueName: \"kubernetes.io/projected/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-kube-api-access-tz6mk\") pod \"network-metrics-daemon-qvs98\" (UID: \"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd\") " pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:29.585467 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:29.585431 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:53:29.585467 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:29.585462 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:53:29.585609 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:29.585475 2577 projected.go:196] Error preparing data for projected volume kube-api-access-9nqkz for pod openshift-network-diagnostics/network-check-target-g584t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:29.585609 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:29.585530 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz podName:8b4fd8cb-a92d-40b2-8133-b04c126332e7 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:30.085512411 +0000 UTC m=+3.064205214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9nqkz" (UniqueName: "kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz") pod "network-check-target-g584t" (UID: "8b4fd8cb-a92d-40b2-8133-b04c126332e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:29.587332 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.587306 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7dwl\" (UniqueName: \"kubernetes.io/projected/864715b0-07ec-483a-8401-089c747422cb-kube-api-access-w7dwl\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.588340 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.588311 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vkxp\" (UniqueName: \"kubernetes.io/projected/535eda1f-d9e7-4908-b958-56f531b68fbb-kube-api-access-2vkxp\") pod \"node-resolver-jz62m\" (UID: \"535eda1f-d9e7-4908-b958-56f531b68fbb\") " pod="openshift-dns/node-resolver-jz62m" Oct 30 11:53:29.589847 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.589592 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbv7k\" (UniqueName: \"kubernetes.io/projected/52ede1dc-817e-434c-874a-2db2a9c10f52-kube-api-access-kbv7k\") pod \"iptables-alerter-s96m4\" (UID: \"52ede1dc-817e-434c-874a-2db2a9c10f52\") " pod="openshift-network-operator/iptables-alerter-s96m4" Oct 30 11:53:29.589847 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.589613 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw9h2\" (UniqueName: \"kubernetes.io/projected/27f694f1-e1de-483f-a5c5-250f52dae768-kube-api-access-bw9h2\") pod \"node-ca-f64ck\" (UID: \"27f694f1-e1de-483f-a5c5-250f52dae768\") " pod="openshift-image-registry/node-ca-f64ck" Oct 30 11:53:29.589847 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.589638 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brmjx\" (UniqueName: \"kubernetes.io/projected/1c3ac2bd-4472-4779-8ca5-460160cdedbb-kube-api-access-brmjx\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.591904 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.591884 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzcmw\" (UniqueName: \"kubernetes.io/projected/fc833d39-b781-486b-9fcc-549cba555ed9-kube-api-access-hzcmw\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.597427 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.597388 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:53:29.694673 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.694636 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:29.697781 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.697595 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gshq6" Oct 30 11:53:29.701910 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.701888 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:53:29.702996 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.702978 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jz62m" Oct 30 11:53:29.712440 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.712418 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7lb7d" Oct 30 11:53:29.720035 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.720018 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f64ck" Oct 30 11:53:29.741896 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:29.741855 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3ac2bd_4472_4779_8ca5_460160cdedbb.slice/crio-5b39fd27f30cf3b4ecfe2293c4990fce79257e8dbe5e18cdcb6d41af03f02b92 WatchSource:0}: Error finding container 5b39fd27f30cf3b4ecfe2293c4990fce79257e8dbe5e18cdcb6d41af03f02b92: Status 404 returned error can't find the container with id 5b39fd27f30cf3b4ecfe2293c4990fce79257e8dbe5e18cdcb6d41af03f02b92 Oct 30 11:53:29.742776 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.742737 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s96m4" Oct 30 11:53:29.744900 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.744867 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:53:29.745644 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:29.745620 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f694f1_e1de_483f_a5c5_250f52dae768.slice/crio-2ac4c9518583c4ba4571dd43309a5be645d0edb1cc1a0e0e93bc0fe8c7a90df7 WatchSource:0}: Error finding container 2ac4c9518583c4ba4571dd43309a5be645d0edb1cc1a0e0e93bc0fe8c7a90df7: Status 404 returned error can't find the container with id 2ac4c9518583c4ba4571dd43309a5be645d0edb1cc1a0e0e93bc0fe8c7a90df7 Oct 30 11:53:29.747397 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:29.747359 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:53:29.751967 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:29.750426 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864715b0_07ec_483a_8401_089c747422cb.slice/crio-fcdbe60b8e623618932dfe168463e773fa7f2ede0126139390f7f77cfeefaaf5 WatchSource:0}: Error finding container fcdbe60b8e623618932dfe168463e773fa7f2ede0126139390f7f77cfeefaaf5: Status 404 returned error can't find the container with id fcdbe60b8e623618932dfe168463e773fa7f2ede0126139390f7f77cfeefaaf5 Oct 30 11:53:29.757907 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:29.757886 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3afa91_aff6_455b_aeda_7275cd8ac8e0.slice/crio-5e77dbd0b1b7ea3f480213918525725419f670a27da07410f16371b00928f042 WatchSource:0}: Error finding container 5e77dbd0b1b7ea3f480213918525725419f670a27da07410f16371b00928f042: Status 404 returned error can't find the container with id 5e77dbd0b1b7ea3f480213918525725419f670a27da07410f16371b00928f042 Oct 30 11:53:29.764597 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:29.764572 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc833d39_b781_486b_9fcc_549cba555ed9.slice/crio-61f2b42ed78030fafb1fef1424e3fe8f595f7d3448273862113f020fc1059f2c WatchSource:0}: Error finding container 61f2b42ed78030fafb1fef1424e3fe8f595f7d3448273862113f020fc1059f2c: Status 404 returned error can't find the container with id 61f2b42ed78030fafb1fef1424e3fe8f595f7d3448273862113f020fc1059f2c Oct 30 11:53:29.766073 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:29.766044 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ede1dc_817e_434c_874a_2db2a9c10f52.slice/crio-81192aec7523c3e2ed796b94d26f534c588906851cbf0bef6f3279b0761b7bf2 WatchSource:0}: Error finding container 81192aec7523c3e2ed796b94d26f534c588906851cbf0bef6f3279b0761b7bf2: Status 404 returned error can't find the container with id 81192aec7523c3e2ed796b94d26f534c588906851cbf0bef6f3279b0761b7bf2 Oct 30 11:53:29.770421 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:29.770391 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedbd4ad4_8205_460e_9385_3151a16e7b68.slice/crio-e6ad89aef93286ff606e8576d07abe2ef749e69c10bcc5d88c91e1e9e65c000e WatchSource:0}: Error finding container e6ad89aef93286ff606e8576d07abe2ef749e69c10bcc5d88c91e1e9e65c000e: Status 404 returned error can't find the container with id e6ad89aef93286ff606e8576d07abe2ef749e69c10bcc5d88c91e1e9e65c000e Oct 30 11:53:30.060568 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:30.060537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:30.060687 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:30.060597 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs\") pod \"network-metrics-daemon-qvs98\" (UID: \"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd\") " pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:30.060748 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:30.060703 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:30.060800 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:30.060741 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:30.060800 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:30.060761 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs podName:4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd nodeName:}" failed. No retries permitted until 2026-10-30 11:53:31.06074403 +0000 UTC m=+4.039436822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs") pod "network-metrics-daemon-qvs98" (UID: "4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:30.060911 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:30.060803 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret podName:68aaec5d-0bc6-4e8b-a311-1797a055bb7b nodeName:}" failed. No retries permitted until 2026-10-30 11:53:31.060785096 +0000 UTC m=+4.039477890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret") pod "global-pull-secret-syncer-pnvz9" (UID: "68aaec5d-0bc6-4e8b-a311-1797a055bb7b") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:30.161906 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:30.161867 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nqkz\" (UniqueName: \"kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz\") pod \"network-check-target-g584t\" (UID: \"8b4fd8cb-a92d-40b2-8133-b04c126332e7\") " pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:30.162148 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:30.162071 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:53:30.162148 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:30.162096 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:53:30.162148 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:30.162110 2577 projected.go:196] Error preparing data for projected volume kube-api-access-9nqkz for pod openshift-network-diagnostics/network-check-target-g584t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:30.162373 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:30.162155 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz podName:8b4fd8cb-a92d-40b2-8133-b04c126332e7 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:31.162141632 +0000 UTC m=+4.140834409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9nqkz" (UniqueName: "kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz") pod "network-check-target-g584t" (UID: "8b4fd8cb-a92d-40b2-8133-b04c126332e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:30.413825 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:30.413474 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 11:48:30 +0000 UTC" deadline="2028-07-21 15:47:18.4455214 +0000 UTC" Oct 30 11:53:30.413825 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:30.413543 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15123h53m48.031983532s" Oct 30 11:53:30.427234 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:30.427198 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" event={"ID":"8314998578de81aba897c7d170293e92","Type":"ContainerStarted","Data":"864076f4ae7bae8cf58ceb926c5533e31c730d2280ec8f65cbeb9655711967f0"} Oct 30 11:53:30.430375 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:30.430345 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" event={"ID":"fc833d39-b781-486b-9fcc-549cba555ed9","Type":"ContainerStarted","Data":"61f2b42ed78030fafb1fef1424e3fe8f595f7d3448273862113f020fc1059f2c"} Oct 30 11:53:30.435344 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:30.432174 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s96m4" event={"ID":"52ede1dc-817e-434c-874a-2db2a9c10f52","Type":"ContainerStarted","Data":"81192aec7523c3e2ed796b94d26f534c588906851cbf0bef6f3279b0761b7bf2"} Oct 30 11:53:30.435344 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:30.434761 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7lb7d" event={"ID":"1a3afa91-aff6-455b-aeda-7275cd8ac8e0","Type":"ContainerStarted","Data":"5e77dbd0b1b7ea3f480213918525725419f670a27da07410f16371b00928f042"} Oct 30 11:53:30.438585 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:30.438516 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gshq6" event={"ID":"864715b0-07ec-483a-8401-089c747422cb","Type":"ContainerStarted","Data":"fcdbe60b8e623618932dfe168463e773fa7f2ede0126139390f7f77cfeefaaf5"} Oct 30 11:53:30.446530 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:30.446504 2577 generic.go:356] "Generic (PLEG): container finished" podID="3e9967fa0ac01d9b4f8c399a65ef8efb" containerID="f8e5a280c6002d9ccb9d53c330f406f1fe5f8714903135b2ef2f4e517d5f9b4e" exitCode=0 Oct 30 11:53:30.446938 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:30.446660 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" event={"ID":"3e9967fa0ac01d9b4f8c399a65ef8efb","Type":"ContainerDied","Data":"f8e5a280c6002d9ccb9d53c330f406f1fe5f8714903135b2ef2f4e517d5f9b4e"} Oct 30 11:53:30.455520 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:30.455492 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hchnk" event={"ID":"edbd4ad4-8205-460e-9385-3151a16e7b68","Type":"ContainerStarted","Data":"e6ad89aef93286ff606e8576d07abe2ef749e69c10bcc5d88c91e1e9e65c000e"} Oct 30 11:53:30.462999 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:30.462928 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jz62m" event={"ID":"535eda1f-d9e7-4908-b958-56f531b68fbb","Type":"ContainerStarted","Data":"052f9555db593197a01666688c3a2781a42079e966f27299450d6fbc934d7f33"} Oct 30 11:53:30.470865 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:30.470789 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" podStartSLOduration=1.470767596 podStartE2EDuration="1.470767596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:30.469710677 +0000 UTC m=+3.448403498" watchObservedRunningTime="2026-10-30 11:53:30.470767596 +0000 UTC m=+3.449460401" Oct 30 11:53:30.483140 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:30.483111 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" event={"ID":"e2b6ec57-b883-408a-ab01-abd192572510","Type":"ContainerStarted","Data":"ec4ae230280ca3268fe9fc538f84aae697eea73bf14c363e17a60963812925ad"} Oct 30 11:53:30.499843 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:30.499784 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f64ck" event={"ID":"27f694f1-e1de-483f-a5c5-250f52dae768","Type":"ContainerStarted","Data":"2ac4c9518583c4ba4571dd43309a5be645d0edb1cc1a0e0e93bc0fe8c7a90df7"} Oct 30 11:53:30.508402 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:30.508111 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" event={"ID":"1c3ac2bd-4472-4779-8ca5-460160cdedbb","Type":"ContainerStarted","Data":"5b39fd27f30cf3b4ecfe2293c4990fce79257e8dbe5e18cdcb6d41af03f02b92"} Oct 30 11:53:31.075437 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:31.075400 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs\") pod \"network-metrics-daemon-qvs98\" (UID: \"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd\") " pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:31.075614 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:31.075472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:31.075693 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:31.075630 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:31.075772 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:31.075692 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret podName:68aaec5d-0bc6-4e8b-a311-1797a055bb7b nodeName:}" failed. No retries permitted until 2026-10-30 11:53:33.075672598 +0000 UTC m=+6.054365379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret") pod "global-pull-secret-syncer-pnvz9" (UID: "68aaec5d-0bc6-4e8b-a311-1797a055bb7b") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:31.076385 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:31.076218 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:31.076385 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:31.076295 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs podName:4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd nodeName:}" failed. No retries permitted until 2026-10-30 11:53:33.076277344 +0000 UTC m=+6.054970129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs") pod "network-metrics-daemon-qvs98" (UID: "4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:31.176093 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:31.176047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nqkz\" (UniqueName: \"kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz\") pod \"network-check-target-g584t\" (UID: \"8b4fd8cb-a92d-40b2-8133-b04c126332e7\") " pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:31.176293 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:31.176265 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:53:31.176498 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:31.176308 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:53:31.176498 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:31.176325 2577 projected.go:196] Error preparing data for projected volume kube-api-access-9nqkz for pod openshift-network-diagnostics/network-check-target-g584t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:31.176498 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:31.176393 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz podName:8b4fd8cb-a92d-40b2-8133-b04c126332e7 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:33.17637221 +0000 UTC m=+6.155065008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9nqkz" (UniqueName: "kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz") pod "network-check-target-g584t" (UID: "8b4fd8cb-a92d-40b2-8133-b04c126332e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:31.411283 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:31.410144 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:31.411283 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:31.410264 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g584t" podUID="8b4fd8cb-a92d-40b2-8133-b04c126332e7" Oct 30 11:53:31.411283 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:31.410698 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:31.411283 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:31.410787 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnvz9" podUID="68aaec5d-0bc6-4e8b-a311-1797a055bb7b" Oct 30 11:53:31.411283 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:31.410853 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:31.411283 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:31.410933 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvs98" podUID="4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd" Oct 30 11:53:31.414399 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:31.414315 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 11:48:30 +0000 UTC" deadline="2028-07-10 01:03:29.061933584 +0000 UTC" Oct 30 11:53:31.414399 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:31.414375 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14845h9m57.647563477s" Oct 30 11:53:31.529227 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:31.528966 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" event={"ID":"3e9967fa0ac01d9b4f8c399a65ef8efb","Type":"ContainerStarted","Data":"bad257fab8bc8ef59d6258631ff37ef3fb6e9004612587df680aa1ec5dcbe9a5"} Oct 30 11:53:33.098489 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:33.098406 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:33.098958 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:33.098505 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs\") pod \"network-metrics-daemon-qvs98\" (UID: \"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd\") " pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:33.098958 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:33.098653 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:33.098958 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:33.098717 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs podName:4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd nodeName:}" failed. No retries permitted until 2026-10-30 11:53:37.098697649 +0000 UTC m=+10.077390452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs") pod "network-metrics-daemon-qvs98" (UID: "4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:33.099179 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:33.099157 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:33.099253 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:33.099215 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret podName:68aaec5d-0bc6-4e8b-a311-1797a055bb7b nodeName:}" failed. No retries permitted until 2026-10-30 11:53:37.099199525 +0000 UTC m=+10.077892308 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret") pod "global-pull-secret-syncer-pnvz9" (UID: "68aaec5d-0bc6-4e8b-a311-1797a055bb7b") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:33.200027 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:33.199402 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nqkz\" (UniqueName: \"kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz\") pod \"network-check-target-g584t\" (UID: \"8b4fd8cb-a92d-40b2-8133-b04c126332e7\") " pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:33.200027 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:33.199609 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:53:33.200027 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:33.199627 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:53:33.200027 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:33.199637 2577 projected.go:196] Error preparing data for projected volume kube-api-access-9nqkz for pod openshift-network-diagnostics/network-check-target-g584t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:33.200027 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:33.199692 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz podName:8b4fd8cb-a92d-40b2-8133-b04c126332e7 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:37.199674182 +0000 UTC m=+10.178366979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9nqkz" (UniqueName: "kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz") pod "network-check-target-g584t" (UID: "8b4fd8cb-a92d-40b2-8133-b04c126332e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:33.410913 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:33.410840 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:33.411086 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:33.410998 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g584t" podUID="8b4fd8cb-a92d-40b2-8133-b04c126332e7" Oct 30 11:53:33.411437 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:33.411416 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:33.411800 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:33.411522 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnvz9" podUID="68aaec5d-0bc6-4e8b-a311-1797a055bb7b" Oct 30 11:53:33.411800 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:33.411586 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:33.411800 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:33.411685 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvs98" podUID="4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd" Oct 30 11:53:33.539756 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:33.538853 2577 generic.go:356] "Generic (PLEG): container finished" podID="edbd4ad4-8205-460e-9385-3151a16e7b68" containerID="30609759de1232481a500ece957c0d7ced169b67c5e10d98044ed22480d668a0" exitCode=0 Oct 30 11:53:33.539756 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:33.538900 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hchnk" event={"ID":"edbd4ad4-8205-460e-9385-3151a16e7b68","Type":"ContainerDied","Data":"30609759de1232481a500ece957c0d7ced169b67c5e10d98044ed22480d668a0"} Oct 30 11:53:33.616946 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:33.615859 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" podStartSLOduration=4.6158394640000004 podStartE2EDuration="4.615839464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:31.572890947 +0000 UTC m=+4.551583750" watchObservedRunningTime="2026-10-30 11:53:33.615839464 +0000 UTC m=+6.594532270" Oct 30 11:53:34.545425 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:34.545378 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7lb7d" event={"ID":"1a3afa91-aff6-455b-aeda-7275cd8ac8e0","Type":"ContainerStarted","Data":"cadbb49b01d0e92846de6e9e0083169f6c1b4e6b23462de7a3a77b9adecf9199"} Oct 30 11:53:35.411628 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:35.410453 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:35.411628 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:35.410593 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g584t" podUID="8b4fd8cb-a92d-40b2-8133-b04c126332e7" Oct 30 11:53:35.411628 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:35.411013 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:35.411628 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:35.411114 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnvz9" podUID="68aaec5d-0bc6-4e8b-a311-1797a055bb7b" Oct 30 11:53:35.411628 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:35.411176 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:35.411628 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:35.411247 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvs98" podUID="4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd" Oct 30 11:53:35.553041 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:35.552969 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f64ck" event={"ID":"27f694f1-e1de-483f-a5c5-250f52dae768","Type":"ContainerStarted","Data":"11527b321b79fbda57c053c768ec38c1901a9310607b964814aea8f4be28d9ef"} Oct 30 11:53:35.594714 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:35.594665 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7lb7d" podStartSLOduration=3.8407323 podStartE2EDuration="7.594650681s" totalImagesPullingTime="3.753918381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:34.590656229 +0000 UTC m=+7.569349031" watchObservedRunningTime="2026-10-30 11:53:35.594650681 +0000 UTC m=+8.573343482" Oct 30 11:53:36.558730 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:36.558634 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" event={"ID":"e2b6ec57-b883-408a-ab01-abd192572510","Type":"ContainerStarted","Data":"3dd58651c89aad561de42650b0f82ac8c803d13a88d40db4f2dbd9674e676873"} Oct 30 11:53:37.041461 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:37.040629 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7lb7d" Oct 30 11:53:37.042654 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:37.042391 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7lb7d" Oct 30 11:53:37.086833 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:37.086713 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f64ck" podStartSLOduration=4.203577431 podStartE2EDuration="9.086690164s" totalImagesPullingTime="4.883112733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:35.595545155 +0000 UTC m=+8.574237959" watchObservedRunningTime="2026-10-30 11:53:37.086690164 +0000 UTC m=+10.065382967" Oct 30 11:53:37.139490 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:37.138459 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs\") pod \"network-metrics-daemon-qvs98\" (UID: \"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd\") " pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:37.139490 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:37.138536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:37.139490 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:37.138674 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:37.139490 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:37.138742 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret podName:68aaec5d-0bc6-4e8b-a311-1797a055bb7b nodeName:}" failed. No retries permitted until 2026-10-30 11:53:45.138722629 +0000 UTC m=+18.117415433 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret") pod "global-pull-secret-syncer-pnvz9" (UID: "68aaec5d-0bc6-4e8b-a311-1797a055bb7b") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:37.139490 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:37.139390 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:37.139490 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:37.139454 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs podName:4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd nodeName:}" failed. No retries permitted until 2026-10-30 11:53:45.139436529 +0000 UTC m=+18.118129321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs") pod "network-metrics-daemon-qvs98" (UID: "4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:37.239641 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:37.239598 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nqkz\" (UniqueName: \"kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz\") pod \"network-check-target-g584t\" (UID: \"8b4fd8cb-a92d-40b2-8133-b04c126332e7\") " pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:37.239806 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:37.239777 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:53:37.239806 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:37.239797 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:53:37.239928 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:37.239811 2577 projected.go:196] Error preparing data for projected volume kube-api-access-9nqkz for pod openshift-network-diagnostics/network-check-target-g584t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:37.239928 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:37.239867 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz podName:8b4fd8cb-a92d-40b2-8133-b04c126332e7 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:45.239849893 +0000 UTC m=+18.218542691 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9nqkz" (UniqueName: "kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz") pod "network-check-target-g584t" (UID: "8b4fd8cb-a92d-40b2-8133-b04c126332e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:37.280622 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:37.280500 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 11:53:37.411618 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:37.411587 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:37.411810 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:37.411717 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g584t" podUID="8b4fd8cb-a92d-40b2-8133-b04c126332e7" Oct 30 11:53:37.412104 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:37.412085 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:37.412195 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:37.412178 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnvz9" podUID="68aaec5d-0bc6-4e8b-a311-1797a055bb7b" Oct 30 11:53:37.412253 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:37.412227 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:37.412353 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:37.412316 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvs98" podUID="4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd" Oct 30 11:53:37.419878 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:37.419770 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T11:53:37.280520449Z","UUID":"a156cb86-e5a2-4355-9a67-22a40900bc76","Handler":null,"Name":"","Endpoint":""} Oct 30 11:53:37.425669 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:37.425647 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 11:53:37.425779 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:37.425678 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 11:53:37.566775 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:37.566500 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" event={"ID":"e2b6ec57-b883-408a-ab01-abd192572510","Type":"ContainerStarted","Data":"9c367d340bdc48f2dbe1eb04a6a2aa9a449dc3dbe3ef7e84f619817ec17982aa"} Oct 30 11:53:38.573245 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:38.569872 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gshq6" event={"ID":"864715b0-07ec-483a-8401-089c747422cb","Type":"ContainerStarted","Data":"e65ac91000a8de58b87f37df6e0e212b6a3ef1ee2bf8fbb128fa3d23fdf5e70c"} Oct 30 11:53:38.573705 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:38.573445 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hchnk" event={"ID":"edbd4ad4-8205-460e-9385-3151a16e7b68","Type":"ContainerStarted","Data":"ad81f53518fc97beb0249bddf2f7dc7694fc11d429e8197076968900fabeb112"} Oct 30 11:53:38.575704 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:38.575676 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jz62m" event={"ID":"535eda1f-d9e7-4908-b958-56f531b68fbb","Type":"ContainerStarted","Data":"9fa0cb0ca2f5a673cecb0efdddd4a2044db6fa8a327d3f7507eb55b9d1f99a2d"} Oct 30 11:53:38.662279 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:38.662214 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-gshq6" podStartSLOduration=2.620805745 podStartE2EDuration="10.662193516s" totalImagesPullingTime="8.041387771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:38.619076797 +0000 UTC m=+11.597769597" watchObservedRunningTime="2026-10-30 11:53:38.662193516 +0000 UTC m=+11.640886320" Oct 30 11:53:38.712011 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:38.711943 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-jz62m" podStartSLOduration=2.329071171 podStartE2EDuration="10.711927054s" totalImagesPullingTime="8.382855883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:38.662986583 +0000 UTC m=+11.641679385" watchObservedRunningTime="2026-10-30 11:53:38.711927054 +0000 UTC m=+11.690619855" Oct 30 11:53:39.409898 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:39.409064 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:39.409898 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:39.409191 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g584t" podUID="8b4fd8cb-a92d-40b2-8133-b04c126332e7" Oct 30 11:53:39.409898 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:39.409570 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:39.409898 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:39.409678 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnvz9" podUID="68aaec5d-0bc6-4e8b-a311-1797a055bb7b" Oct 30 11:53:39.409898 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:39.409747 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:39.409898 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:39.409830 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvs98" podUID="4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd" Oct 30 11:53:39.580096 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:39.580019 2577 generic.go:356] "Generic (PLEG): container finished" podID="edbd4ad4-8205-460e-9385-3151a16e7b68" containerID="ad81f53518fc97beb0249bddf2f7dc7694fc11d429e8197076968900fabeb112" exitCode=0 Oct 30 11:53:39.580601 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:39.580094 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hchnk" event={"ID":"edbd4ad4-8205-460e-9385-3151a16e7b68","Type":"ContainerDied","Data":"ad81f53518fc97beb0249bddf2f7dc7694fc11d429e8197076968900fabeb112"} Oct 30 11:53:39.583631 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:39.583489 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" event={"ID":"e2b6ec57-b883-408a-ab01-abd192572510","Type":"ContainerStarted","Data":"1c1de264d6ea9964c36e23438f3493f94b31c31bb6458a5f6fe669afdc2d390e"} Oct 30 11:53:40.586619 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:40.586587 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" event={"ID":"fc833d39-b781-486b-9fcc-549cba555ed9","Type":"ContainerStarted","Data":"eeb225babf47cd60554810975d29410613484170a7cea163ae5b5194c234d919"} Oct 30 11:53:40.588100 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:40.588079 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s96m4" event={"ID":"52ede1dc-817e-434c-874a-2db2a9c10f52","Type":"ContainerStarted","Data":"28b280d248b83384b529ee931ab1ab7fea6de325414322a6709e70bddd41cd01"} Oct 30 11:53:40.628574 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:40.628523 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" podStartSLOduration=4.288351346 podStartE2EDuration="12.628507414s" totalImagesPullingTime="8.340156068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:39.694382673 +0000 UTC m=+12.673075472" watchObservedRunningTime="2026-10-30 11:53:40.628507414 +0000 UTC m=+13.607200213" Oct 30 11:53:40.628906 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:40.628885 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" podStartSLOduration=1.963739296 podStartE2EDuration="12.628880627s" totalImagesPullingTime="10.665141331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:40.628436282 +0000 UTC m=+13.607129083" watchObservedRunningTime="2026-10-30 11:53:40.628880627 +0000 UTC m=+13.607573447" Oct 30 11:53:40.670766 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:40.670594 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s96m4" podStartSLOduration=4.149276495 podStartE2EDuration="12.670580992s" totalImagesPullingTime="8.521304497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:40.669878309 +0000 UTC m=+13.648571107" watchObservedRunningTime="2026-10-30 11:53:40.670580992 +0000 UTC m=+13.649273789" Oct 30 11:53:41.408185 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:41.408151 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:41.408385 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:41.408216 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:41.408385 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:41.408225 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:41.408385 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:41.408340 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g584t" podUID="8b4fd8cb-a92d-40b2-8133-b04c126332e7" Oct 30 11:53:41.408539 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:41.408408 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnvz9" podUID="68aaec5d-0bc6-4e8b-a311-1797a055bb7b" Oct 30 11:53:41.408539 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:41.408518 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvs98" podUID="4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd" Oct 30 11:53:41.593269 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:41.593214 2577 generic.go:356] "Generic (PLEG): container finished" podID="edbd4ad4-8205-460e-9385-3151a16e7b68" containerID="363ecfdfc38f12a1827471bf130dfc3cf81020d3f74ae830c5a715e79fdad493" exitCode=0 Oct 30 11:53:41.593733 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:41.593538 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hchnk" event={"ID":"edbd4ad4-8205-460e-9385-3151a16e7b68","Type":"ContainerDied","Data":"363ecfdfc38f12a1827471bf130dfc3cf81020d3f74ae830c5a715e79fdad493"} Oct 30 11:53:42.598850 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:42.598766 2577 generic.go:356] "Generic (PLEG): container finished" podID="edbd4ad4-8205-460e-9385-3151a16e7b68" containerID="84b38ee5b78874b3ba63aea701531b75db4e36b58104379752c623bb528cf2b8" exitCode=0 Oct 30 11:53:42.598850 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:42.598829 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hchnk" event={"ID":"edbd4ad4-8205-460e-9385-3151a16e7b68","Type":"ContainerDied","Data":"84b38ee5b78874b3ba63aea701531b75db4e36b58104379752c623bb528cf2b8"} Oct 30 11:53:43.407876 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:43.407840 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:43.407876 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:43.407848 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:43.408087 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:43.407884 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:43.408087 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:43.407984 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g584t" podUID="8b4fd8cb-a92d-40b2-8133-b04c126332e7" Oct 30 11:53:43.408087 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:43.408050 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnvz9" podUID="68aaec5d-0bc6-4e8b-a311-1797a055bb7b" Oct 30 11:53:43.408191 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:43.408144 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvs98" podUID="4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd" Oct 30 11:53:45.197469 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:45.197295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs\") pod \"network-metrics-daemon-qvs98\" (UID: \"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd\") " pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:45.197975 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:45.197502 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:45.197975 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:45.197435 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:45.197975 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:45.197600 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs podName:4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd nodeName:}" failed. No retries permitted until 2026-10-30 11:54:01.197585012 +0000 UTC m=+34.176277792 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs") pod "network-metrics-daemon-qvs98" (UID: "4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 11:53:45.197975 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:45.197600 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:45.197975 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:45.197646 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret podName:68aaec5d-0bc6-4e8b-a311-1797a055bb7b nodeName:}" failed. No retries permitted until 2026-10-30 11:54:01.197629994 +0000 UTC m=+34.176322771 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret") pod "global-pull-secret-syncer-pnvz9" (UID: "68aaec5d-0bc6-4e8b-a311-1797a055bb7b") : object "kube-system"/"original-pull-secret" not registered Oct 30 11:53:45.297939 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:45.297907 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nqkz\" (UniqueName: \"kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz\") pod \"network-check-target-g584t\" (UID: \"8b4fd8cb-a92d-40b2-8133-b04c126332e7\") " pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:45.298066 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:45.298049 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 11:53:45.298106 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:45.298070 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 11:53:45.298106 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:45.298079 2577 projected.go:196] Error preparing data for projected volume kube-api-access-9nqkz for pod openshift-network-diagnostics/network-check-target-g584t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:45.298159 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:45.298124 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz podName:8b4fd8cb-a92d-40b2-8133-b04c126332e7 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:01.298111162 +0000 UTC m=+34.276803938 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9nqkz" (UniqueName: "kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz") pod "network-check-target-g584t" (UID: "8b4fd8cb-a92d-40b2-8133-b04c126332e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 11:53:45.407950 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:45.407914 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:45.408105 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:45.407948 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:45.408105 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:45.408010 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnvz9" podUID="68aaec5d-0bc6-4e8b-a311-1797a055bb7b" Oct 30 11:53:45.408105 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:45.408048 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:45.408214 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:45.408133 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvs98" podUID="4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd" Oct 30 11:53:45.408268 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:45.408226 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g584t" podUID="8b4fd8cb-a92d-40b2-8133-b04c126332e7" Oct 30 11:53:45.607384 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:45.607312 2577 generic.go:356] "Generic (PLEG): container finished" podID="edbd4ad4-8205-460e-9385-3151a16e7b68" containerID="0bb8663f233466d320137adf037d81661cc1c452909e3fc4a7d04bbd581c1db5" exitCode=0 Oct 30 11:53:45.607519 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:45.607396 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hchnk" event={"ID":"edbd4ad4-8205-460e-9385-3151a16e7b68","Type":"ContainerDied","Data":"0bb8663f233466d320137adf037d81661cc1c452909e3fc4a7d04bbd581c1db5"} Oct 30 11:53:45.610384 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:45.610362 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" event={"ID":"1c3ac2bd-4472-4779-8ca5-460160cdedbb","Type":"ContainerStarted","Data":"b96e0fc298c98bd02b19cb780df083c00887cec42e2e4ba4f69c83176b5e3f66"} Oct 30 11:53:45.610432 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:45.610418 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" event={"ID":"1c3ac2bd-4472-4779-8ca5-460160cdedbb","Type":"ContainerStarted","Data":"d3b9c7cab01e9862961ff3b4cdfa62586e6db4ba00bfe6fd6c0ccf84bf7fcc4b"} Oct 30 11:53:45.610476 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:45.610446 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" event={"ID":"1c3ac2bd-4472-4779-8ca5-460160cdedbb","Type":"ContainerStarted","Data":"aa5a716ae52904393400be2ec405ce9ac70e70cec1aa08fd145fe3de13b31a71"} Oct 30 11:53:45.610476 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:45.610473 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" event={"ID":"1c3ac2bd-4472-4779-8ca5-460160cdedbb","Type":"ContainerStarted","Data":"161e271a319ab9604ba0863f529cf1474b8edcb40a2608c4ba5087b1642b489b"} Oct 30 11:53:45.610533 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:45.610500 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" event={"ID":"1c3ac2bd-4472-4779-8ca5-460160cdedbb","Type":"ContainerStarted","Data":"5f2f2e48227fd0f5899028739ef1ab7442b03e82d46de10365b78d4d57b611d3"} Oct 30 11:53:45.610533 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:45.610525 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" event={"ID":"1c3ac2bd-4472-4779-8ca5-460160cdedbb","Type":"ContainerStarted","Data":"fe2697188072876416bb719f95bbff35d72a433d668d2070acdf5181bdd95f3f"} Oct 30 11:53:46.614585 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:46.614551 2577 generic.go:356] "Generic (PLEG): container finished" podID="edbd4ad4-8205-460e-9385-3151a16e7b68" containerID="5bbe5e23362fdd4cc5bea195b335b40c011ddc779695356a74e4f31dfee003c2" exitCode=0 Oct 30 11:53:46.614933 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:46.614617 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hchnk" event={"ID":"edbd4ad4-8205-460e-9385-3151a16e7b68","Type":"ContainerDied","Data":"5bbe5e23362fdd4cc5bea195b335b40c011ddc779695356a74e4f31dfee003c2"} Oct 30 11:53:47.408140 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:47.407965 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:47.408311 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:47.407967 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:47.408311 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:47.408210 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g584t" podUID="8b4fd8cb-a92d-40b2-8133-b04c126332e7" Oct 30 11:53:47.408311 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:47.407997 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:47.408311 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:47.408291 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnvz9" podUID="68aaec5d-0bc6-4e8b-a311-1797a055bb7b" Oct 30 11:53:47.408530 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:47.408372 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvs98" podUID="4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd" Oct 30 11:53:47.620170 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:47.620133 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hchnk" event={"ID":"edbd4ad4-8205-460e-9385-3151a16e7b68","Type":"ContainerStarted","Data":"0c06fcb6ce1ad70aeaeea9bc61f66196c3c63d05e758740aedec7b5e2fa26d49"} Oct 30 11:53:47.623260 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:47.623228 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" event={"ID":"1c3ac2bd-4472-4779-8ca5-460160cdedbb","Type":"ContainerStarted","Data":"ac980aab8755784e81617ef599a08421cb2f8ceed9da890b81cdbcf99bed4446"} Oct 30 11:53:47.692833 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:47.692765 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hchnk" podStartSLOduration=8.308616255 podStartE2EDuration="19.692755818s" totalImagesPullingTime="11.384139563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:47.692421634 +0000 UTC m=+20.671114432" watchObservedRunningTime="2026-10-30 11:53:47.692755818 +0000 UTC m=+20.671448616" Oct 30 11:53:49.407946 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:49.407916 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:49.408565 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:49.407946 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:49.408565 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:49.408068 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnvz9" podUID="68aaec5d-0bc6-4e8b-a311-1797a055bb7b" Oct 30 11:53:49.408565 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:49.408100 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:49.408565 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:49.408232 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g584t" podUID="8b4fd8cb-a92d-40b2-8133-b04c126332e7" Oct 30 11:53:49.408565 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:49.408342 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvs98" podUID="4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd" Oct 30 11:53:49.630289 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:49.630241 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" event={"ID":"1c3ac2bd-4472-4779-8ca5-460160cdedbb","Type":"ContainerStarted","Data":"c20ee826140e337f3e2fa76ff01b1a94454f1eb45e7318334553c0d53c8886c3"} Oct 30 11:53:49.630624 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:49.630603 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:49.630734 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:49.630667 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:49.645694 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:49.645674 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:49.645818 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:49.645797 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:49.721264 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:49.721192 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" podStartSLOduration=6.706437803 podStartE2EDuration="21.721177406s" totalImagesPullingTime="15.014739603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:49.709612477 +0000 UTC m=+22.688305275" watchObservedRunningTime="2026-10-30 11:53:49.721177406 +0000 UTC m=+22.699870204" Oct 30 11:53:50.632356 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:50.632318 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 11:53:51.310923 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:51.310851 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pnvz9"] Oct 30 11:53:51.311048 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:51.310983 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:51.311089 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:51.311058 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnvz9" podUID="68aaec5d-0bc6-4e8b-a311-1797a055bb7b" Oct 30 11:53:51.314311 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:51.314287 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvs98"] Oct 30 11:53:51.314419 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:51.314390 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:51.314501 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:51.314468 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvs98" podUID="4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd" Oct 30 11:53:51.323173 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:51.323157 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g584t"] Oct 30 11:53:51.323244 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:51.323234 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:51.323337 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:51.323323 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g584t" podUID="8b4fd8cb-a92d-40b2-8133-b04c126332e7" Oct 30 11:53:51.634940 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:51.634913 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 11:53:52.784110 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:52.784062 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:53:52.985671 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:52.985637 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:53:53.407528 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:53.407498 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:53.407803 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:53.407516 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:53.407803 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:53.407585 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:53.407989 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:53.407901 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g584t" podUID="8b4fd8cb-a92d-40b2-8133-b04c126332e7" Oct 30 11:53:53.408053 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:53.408009 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnvz9" podUID="68aaec5d-0bc6-4e8b-a311-1797a055bb7b" Oct 30 11:53:53.408302 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:53.408230 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvs98" podUID="4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd" Oct 30 11:53:55.408309 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:55.408124 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:55.408744 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:55.408130 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:55.408744 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:55.408378 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g584t" podUID="8b4fd8cb-a92d-40b2-8133-b04c126332e7" Oct 30 11:53:55.408744 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:55.408476 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnvz9" podUID="68aaec5d-0bc6-4e8b-a311-1797a055bb7b" Oct 30 11:53:55.408744 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:55.408141 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:55.408744 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:55.408591 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvs98" podUID="4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd" Oct 30 11:53:56.663790 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:56.663755 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:53:56.786169 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:56.786140 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:53:57.409113 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.409080 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:57.409330 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.409202 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:57.409330 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:57.409214 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvs98" podUID="4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd" Oct 30 11:53:57.409330 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:57.409279 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g584t" podUID="8b4fd8cb-a92d-40b2-8133-b04c126332e7" Oct 30 11:53:57.409330 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.409304 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:57.409554 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:57.409395 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pnvz9" podUID="68aaec5d-0bc6-4e8b-a311-1797a055bb7b" Oct 30 11:53:57.757873 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.757847 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeReady" Oct 30 11:53:57.758285 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.757990 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 11:53:57.803946 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.803917 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:53:57.816918 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.816893 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lfqp6"] Oct 30 11:53:57.819771 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.819757 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lfqp6" Oct 30 11:53:57.824372 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.824356 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lqnn8"] Oct 30 11:53:57.827164 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.827150 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lqnn8" Oct 30 11:53:57.835666 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.835644 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 11:53:57.835666 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.835644 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 11:53:57.835820 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.835699 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 11:53:57.835820 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.835779 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tdxpc\"" Oct 30 11:53:57.840028 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.840011 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lfqp6"] Oct 30 11:53:57.847890 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.847871 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lqnn8"] Oct 30 11:53:57.932938 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.932911 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:53:57.938940 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.938915 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 11:53:57.938940 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.938910 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 11:53:57.939190 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.939178 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjl8c\"" Oct 30 11:53:57.993386 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.993367 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2vgd\" (UniqueName: \"kubernetes.io/projected/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-kube-api-access-m2vgd\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:53:57.993453 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.993393 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-tmp-dir\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:53:57.993453 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.993412 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:53:57.993552 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.993537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/125e1073-73c6-4aae-9c1b-4dc6e047e241-cert\") pod \"ingress-canary-lfqp6\" (UID: \"125e1073-73c6-4aae-9c1b-4dc6e047e241\") " pod="openshift-ingress-canary/ingress-canary-lfqp6" Oct 30 11:53:57.993593 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.993579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5wc8\" (UniqueName: \"kubernetes.io/projected/125e1073-73c6-4aae-9c1b-4dc6e047e241-kube-api-access-h5wc8\") pod \"ingress-canary-lfqp6\" (UID: \"125e1073-73c6-4aae-9c1b-4dc6e047e241\") " pod="openshift-ingress-canary/ingress-canary-lfqp6" Oct 30 11:53:57.993623 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:57.993607 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-config-volume\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:53:58.094090 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.094027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-tmp-dir\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:53:58.094090 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.094060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:53:58.094211 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:58.094161 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:53:58.094211 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.094192 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/125e1073-73c6-4aae-9c1b-4dc6e047e241-cert\") pod \"ingress-canary-lfqp6\" (UID: \"125e1073-73c6-4aae-9c1b-4dc6e047e241\") " pod="openshift-ingress-canary/ingress-canary-lfqp6" Oct 30 11:53:58.094556 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:58.094531 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls podName:62adf1a5-f87e-4cbc-86e5-bcd2cefbd390 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:58.59450573 +0000 UTC m=+31.573198512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls") pod "dns-default-lqnn8" (UID: "62adf1a5-f87e-4cbc-86e5-bcd2cefbd390") : secret "dns-default-metrics-tls" not found Oct 30 11:53:58.094556 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.094547 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5wc8\" (UniqueName: \"kubernetes.io/projected/125e1073-73c6-4aae-9c1b-4dc6e047e241-kube-api-access-h5wc8\") pod \"ingress-canary-lfqp6\" (UID: \"125e1073-73c6-4aae-9c1b-4dc6e047e241\") " pod="openshift-ingress-canary/ingress-canary-lfqp6" Oct 30 11:53:58.094681 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.094601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-config-volume\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:53:58.094681 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.094630 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2vgd\" (UniqueName: \"kubernetes.io/projected/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-kube-api-access-m2vgd\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:53:58.094681 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.094646 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-tmp-dir\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:53:58.095214 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.095187 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-config-volume\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:53:58.098864 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.098844 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/125e1073-73c6-4aae-9c1b-4dc6e047e241-cert\") pod \"ingress-canary-lfqp6\" (UID: \"125e1073-73c6-4aae-9c1b-4dc6e047e241\") " pod="openshift-ingress-canary/ingress-canary-lfqp6" Oct 30 11:53:58.118645 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.118618 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2vgd\" (UniqueName: \"kubernetes.io/projected/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-kube-api-access-m2vgd\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:53:58.118789 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.118768 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5wc8\" (UniqueName: \"kubernetes.io/projected/125e1073-73c6-4aae-9c1b-4dc6e047e241-kube-api-access-h5wc8\") pod \"ingress-canary-lfqp6\" (UID: \"125e1073-73c6-4aae-9c1b-4dc6e047e241\") " pod="openshift-ingress-canary/ingress-canary-lfqp6" Oct 30 11:53:58.131508 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.131492 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lfqp6" Oct 30 11:53:58.265936 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.265900 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7lb7d" Oct 30 11:53:58.266116 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.266080 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 11:53:58.266548 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.266518 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7lb7d" Oct 30 11:53:58.386063 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.386020 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:53:58.444909 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.444889 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lfqp6"] Oct 30 11:53:58.448094 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:53:58.448072 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125e1073_73c6_4aae_9c1b_4dc6e047e241.slice/crio-a090e0699c8024b05ff126f7ca48701b8dc97108a9f5364ed1d6af4a1598521e WatchSource:0}: Error finding container a090e0699c8024b05ff126f7ca48701b8dc97108a9f5364ed1d6af4a1598521e: Status 404 returned error can't find the container with id a090e0699c8024b05ff126f7ca48701b8dc97108a9f5364ed1d6af4a1598521e Oct 30 11:53:58.598707 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.598675 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:53:58.598836 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:58.598788 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:53:58.598889 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:58.598849 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls podName:62adf1a5-f87e-4cbc-86e5-bcd2cefbd390 nodeName:}" failed. No retries permitted until 2026-10-30 11:53:59.598831505 +0000 UTC m=+32.577524296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls") pod "dns-default-lqnn8" (UID: "62adf1a5-f87e-4cbc-86e5-bcd2cefbd390") : secret "dns-default-metrics-tls" not found Oct 30 11:53:58.651263 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.651178 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lfqp6" event={"ID":"125e1073-73c6-4aae-9c1b-4dc6e047e241","Type":"ContainerStarted","Data":"a090e0699c8024b05ff126f7ca48701b8dc97108a9f5364ed1d6af4a1598521e"} Oct 30 11:53:58.905210 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:58.905128 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:53:59.021609 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.021564 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:53:59.170098 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.170027 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:59.170285 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.170269 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 11:53:59.188226 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.188204 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:53:59.381725 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.381693 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:53:59.384237 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.384217 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7r9dz"] Oct 30 11:53:59.420010 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.419981 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7r9dz"] Oct 30 11:53:59.420164 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.420076 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7r9dz" Oct 30 11:53:59.420229 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.420209 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:53:59.420297 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.420289 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:53:59.420344 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.420307 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:53:59.506592 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.506562 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhwsn\" (UniqueName: \"kubernetes.io/projected/39ef6d56-2baa-4f93-9268-12a74e0ef903-kube-api-access-fhwsn\") pod \"migrator-66df5d496b-7r9dz\" (UID: \"39ef6d56-2baa-4f93-9268-12a74e0ef903\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7r9dz" Oct 30 11:53:59.518342 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.518315 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bdxnx"] Oct 30 11:53:59.538105 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.538086 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:53:59.538830 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.538790 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zfblq\"" Oct 30 11:53:59.538919 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.538872 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rpjw7\"" Oct 30 11:53:59.538919 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.538876 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 11:53:59.539374 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.539348 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 11:53:59.539600 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.539577 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 11:53:59.539714 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.539699 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jjcjh\"" Oct 30 11:53:59.539997 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.539981 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 11:53:59.541391 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.541372 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 11:53:59.541483 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.541395 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sjprl\"" Oct 30 11:53:59.541483 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.541432 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 11:53:59.546369 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.545896 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:53:59.555083 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.555060 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bdxnx"] Oct 30 11:53:59.607415 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.607385 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:53:59.607586 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:59.607568 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:53:59.607666 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:59.607632 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls podName:62adf1a5-f87e-4cbc-86e5-bcd2cefbd390 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:01.607614173 +0000 UTC m=+34.586306969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls") pod "dns-default-lqnn8" (UID: "62adf1a5-f87e-4cbc-86e5-bcd2cefbd390") : secret "dns-default-metrics-tls" not found Oct 30 11:53:59.607741 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.607660 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhwsn\" (UniqueName: \"kubernetes.io/projected/39ef6d56-2baa-4f93-9268-12a74e0ef903-kube-api-access-fhwsn\") pod \"migrator-66df5d496b-7r9dz\" (UID: \"39ef6d56-2baa-4f93-9268-12a74e0ef903\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7r9dz" Oct 30 11:53:59.640803 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.640768 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 11:53:59.640939 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.640860 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8pg5x\"" Oct 30 11:53:59.641411 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.641376 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 11:53:59.641411 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.641394 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 11:53:59.641735 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.641708 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 11:53:59.643227 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.643201 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhwsn\" (UniqueName: \"kubernetes.io/projected/39ef6d56-2baa-4f93-9268-12a74e0ef903-kube-api-access-fhwsn\") pod \"migrator-66df5d496b-7r9dz\" (UID: \"39ef6d56-2baa-4f93-9268-12a74e0ef903\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7r9dz" Oct 30 11:53:59.708395 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.708316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82e6695b-1b88-4372-9d00-ddf341253c59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:53:59.708545 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.708402 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82e6695b-1b88-4372-9d00-ddf341253c59-data-volume\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:53:59.708545 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.708430 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82e6695b-1b88-4372-9d00-ddf341253c59-crio-socket\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:53:59.708545 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.708456 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:53:59.708545 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.708535 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg275\" (UniqueName: \"kubernetes.io/projected/82e6695b-1b88-4372-9d00-ddf341253c59-kube-api-access-kg275\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:53:59.747477 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.747438 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7r9dz" Oct 30 11:53:59.751981 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.751945 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:53:59.811813 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.811778 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82e6695b-1b88-4372-9d00-ddf341253c59-crio-socket\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:53:59.812010 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.811817 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:53:59.812010 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.811890 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg275\" (UniqueName: \"kubernetes.io/projected/82e6695b-1b88-4372-9d00-ddf341253c59-kube-api-access-kg275\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:53:59.812121 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.812038 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82e6695b-1b88-4372-9d00-ddf341253c59-crio-socket\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:53:59.812121 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:59.812046 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:53:59.812218 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:59.812129 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls podName:82e6695b-1b88-4372-9d00-ddf341253c59 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:00.312106072 +0000 UTC m=+33.290798862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdxnx" (UID: "82e6695b-1b88-4372-9d00-ddf341253c59") : secret "insights-runtime-extractor-tls" not found Oct 30 11:53:59.812394 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.812371 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82e6695b-1b88-4372-9d00-ddf341253c59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:53:59.812466 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.812426 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82e6695b-1b88-4372-9d00-ddf341253c59-data-volume\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:53:59.812722 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:59.812701 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 11:53:59.812796 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:53:59.812761 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82e6695b-1b88-4372-9d00-ddf341253c59-kube-rbac-proxy-cm podName:82e6695b-1b88-4372-9d00-ddf341253c59 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:00.312747907 +0000 UTC m=+33.291440697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82e6695b-1b88-4372-9d00-ddf341253c59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bdxnx" (UID: "82e6695b-1b88-4372-9d00-ddf341253c59") : configmap "kube-rbac-proxy" not found Oct 30 11:53:59.812860 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.812817 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82e6695b-1b88-4372-9d00-ddf341253c59-data-volume\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:53:59.837176 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.837123 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg275\" (UniqueName: \"kubernetes.io/projected/82e6695b-1b88-4372-9d00-ddf341253c59-kube-api-access-kg275\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:53:59.874712 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:53:59.874681 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:00.073818 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:00.073742 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7r9dz"] Oct 30 11:54:00.080899 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:54:00.080869 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ef6d56_2baa_4f93_9268_12a74e0ef903.slice/crio-c69d9dfb491728afa0d6a3d6a46b2802ddfd18ee95dd01558ea41a8dc1ed1ff3 WatchSource:0}: Error finding container c69d9dfb491728afa0d6a3d6a46b2802ddfd18ee95dd01558ea41a8dc1ed1ff3: Status 404 returned error can't find the container with id c69d9dfb491728afa0d6a3d6a46b2802ddfd18ee95dd01558ea41a8dc1ed1ff3 Oct 30 11:54:00.317326 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:00.317295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82e6695b-1b88-4372-9d00-ddf341253c59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:00.317489 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:00.317334 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:00.317489 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:54:00.317442 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 11:54:00.317582 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:54:00.317526 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82e6695b-1b88-4372-9d00-ddf341253c59-kube-rbac-proxy-cm podName:82e6695b-1b88-4372-9d00-ddf341253c59 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:01.317504288 +0000 UTC m=+34.296197085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82e6695b-1b88-4372-9d00-ddf341253c59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bdxnx" (UID: "82e6695b-1b88-4372-9d00-ddf341253c59") : configmap "kube-rbac-proxy" not found Oct 30 11:54:00.317582 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:54:00.317447 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:00.317692 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:54:00.317637 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls podName:82e6695b-1b88-4372-9d00-ddf341253c59 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:01.31761837 +0000 UTC m=+34.296311160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdxnx" (UID: "82e6695b-1b88-4372-9d00-ddf341253c59") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:00.437087 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:00.437054 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:00.561076 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:00.561041 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:00.657974 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:00.657936 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7r9dz" event={"ID":"39ef6d56-2baa-4f93-9268-12a74e0ef903","Type":"ContainerStarted","Data":"c69d9dfb491728afa0d6a3d6a46b2802ddfd18ee95dd01558ea41a8dc1ed1ff3"} Oct 30 11:54:01.224117 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:01.224088 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs\") pod \"network-metrics-daemon-qvs98\" (UID: \"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd\") " pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:54:01.224460 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:01.224144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:54:01.224460 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:54:01.224268 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 11:54:01.224460 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:54:01.224340 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs podName:4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd nodeName:}" failed. No retries permitted until 2026-10-30 11:54:33.224325269 +0000 UTC m=+66.203018048 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs") pod "network-metrics-daemon-qvs98" (UID: "4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd") : secret "metrics-daemon-secret" not found Oct 30 11:54:01.226590 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:01.226559 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:54:01.269179 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:01.269156 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:54:01.325035 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:01.324870 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nqkz\" (UniqueName: \"kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz\") pod \"network-check-target-g584t\" (UID: \"8b4fd8cb-a92d-40b2-8133-b04c126332e7\") " pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:54:01.325035 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:01.324921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82e6695b-1b88-4372-9d00-ddf341253c59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:01.325035 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:01.324952 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:01.325220 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:54:01.325036 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 11:54:01.325220 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:54:01.325100 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82e6695b-1b88-4372-9d00-ddf341253c59-kube-rbac-proxy-cm podName:82e6695b-1b88-4372-9d00-ddf341253c59 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:03.325083925 +0000 UTC m=+36.303776702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82e6695b-1b88-4372-9d00-ddf341253c59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bdxnx" (UID: "82e6695b-1b88-4372-9d00-ddf341253c59") : configmap "kube-rbac-proxy" not found Oct 30 11:54:01.325444 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:54:01.325372 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:01.325444 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:54:01.325422 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls podName:82e6695b-1b88-4372-9d00-ddf341253c59 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:03.325410623 +0000 UTC m=+36.304103399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdxnx" (UID: "82e6695b-1b88-4372-9d00-ddf341253c59") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:01.328798 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:01.328777 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nqkz\" (UniqueName: \"kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz\") pod \"network-check-target-g584t\" (UID: \"8b4fd8cb-a92d-40b2-8133-b04c126332e7\") " pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:54:01.344117 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:01.344088 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:01.602112 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:01.602043 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pnvz9"] Oct 30 11:54:01.610169 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:01.610151 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:54:01.626917 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:01.626900 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:54:01.627056 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:54:01.627042 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:54:01.627103 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:54:01.627095 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls podName:62adf1a5-f87e-4cbc-86e5-bcd2cefbd390 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:05.627081849 +0000 UTC m=+38.605774645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls") pod "dns-default-lqnn8" (UID: "62adf1a5-f87e-4cbc-86e5-bcd2cefbd390") : secret "dns-default-metrics-tls" not found Oct 30 11:54:01.661358 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:01.661326 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lfqp6" event={"ID":"125e1073-73c6-4aae-9c1b-4dc6e047e241","Type":"ContainerStarted","Data":"d85a101f79be5f1c5e357ffd02a737ad349e7d715c301f02b830f48e0e8ded30"} Oct 30 11:54:01.706046 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:01.706002 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lfqp6" podStartSLOduration=1.952052077 podStartE2EDuration="4.705989393s" totalImagesPullingTime="2.753937316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:01.70561069 +0000 UTC m=+34.684303507" watchObservedRunningTime="2026-10-30 11:54:01.705989393 +0000 UTC m=+34.684682185" Oct 30 11:54:01.706181 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:54:01.706099 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68aaec5d_0bc6_4e8b_a311_1797a055bb7b.slice/crio-84dd1f111347bc3a986759810ffd4928000cb6c86746b5216ca22a74cdb5bf83 WatchSource:0}: Error finding container 84dd1f111347bc3a986759810ffd4928000cb6c86746b5216ca22a74cdb5bf83: Status 404 returned error can't find the container with id 84dd1f111347bc3a986759810ffd4928000cb6c86746b5216ca22a74cdb5bf83 Oct 30 11:54:02.027188 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:02.027157 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g584t"] Oct 30 11:54:02.030441 ip-10-0-138-23 kubenswrapper[2577]: W1030 11:54:02.030414 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b4fd8cb_a92d_40b2_8133_b04c126332e7.slice/crio-1345f65fd1b97393ec76019253855fefed59956dae9f3758ee3f67a0a1301d4c WatchSource:0}: Error finding container 1345f65fd1b97393ec76019253855fefed59956dae9f3758ee3f67a0a1301d4c: Status 404 returned error can't find the container with id 1345f65fd1b97393ec76019253855fefed59956dae9f3758ee3f67a0a1301d4c Oct 30 11:54:02.666133 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:02.666095 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g584t" event={"ID":"8b4fd8cb-a92d-40b2-8133-b04c126332e7","Type":"ContainerStarted","Data":"1345f65fd1b97393ec76019253855fefed59956dae9f3758ee3f67a0a1301d4c"} Oct 30 11:54:02.667951 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:02.667917 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7r9dz" event={"ID":"39ef6d56-2baa-4f93-9268-12a74e0ef903","Type":"ContainerStarted","Data":"bf8fd985b62abaf28254713b212f34103a1c4bfab39ea71332ce4ee56c0a18f7"} Oct 30 11:54:02.670017 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:02.669986 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pnvz9" event={"ID":"68aaec5d-0bc6-4e8b-a311-1797a055bb7b","Type":"ContainerStarted","Data":"84dd1f111347bc3a986759810ffd4928000cb6c86746b5216ca22a74cdb5bf83"} Oct 30 11:54:03.340627 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:03.339974 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:03.340627 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:03.340094 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82e6695b-1b88-4372-9d00-ddf341253c59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:03.340627 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:54:03.340136 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:03.340627 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:54:03.340180 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 11:54:03.340627 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:54:03.340223 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls podName:82e6695b-1b88-4372-9d00-ddf341253c59 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:07.34019064 +0000 UTC m=+40.318883419 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdxnx" (UID: "82e6695b-1b88-4372-9d00-ddf341253c59") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:03.340627 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:54:03.340243 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82e6695b-1b88-4372-9d00-ddf341253c59-kube-rbac-proxy-cm podName:82e6695b-1b88-4372-9d00-ddf341253c59 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:07.340235583 +0000 UTC m=+40.318928361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82e6695b-1b88-4372-9d00-ddf341253c59-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bdxnx" (UID: "82e6695b-1b88-4372-9d00-ddf341253c59") : configmap "kube-rbac-proxy" not found Oct 30 11:54:04.330869 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:04.330831 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:04.662509 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:04.662471 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:04.799882 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:04.799847 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:04.921120 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:04.920992 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:05.061264 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:05.061225 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:05.183279 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:05.183181 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:05.597344 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:05.597232 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:05.659229 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:05.659189 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:54:05.659385 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:54:05.659370 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:54:05.659434 ip-10-0-138-23 kubenswrapper[2577]: E1030 11:54:05.659425 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls podName:62adf1a5-f87e-4cbc-86e5-bcd2cefbd390 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:13.659410942 +0000 UTC m=+46.638103718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls") pod "dns-default-lqnn8" (UID: "62adf1a5-f87e-4cbc-86e5-bcd2cefbd390") : secret "dns-default-metrics-tls" not found Oct 30 11:54:05.679324 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:05.679288 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g584t" event={"ID":"8b4fd8cb-a92d-40b2-8133-b04c126332e7","Type":"ContainerStarted","Data":"960b68295bb3b525c83e09b760c115066ab66d187bb0194c6e9c059fa58d3fbe"} Oct 30 11:54:05.679443 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:05.679433 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:54:05.752129 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:05.752067 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7r9dz" podStartSLOduration=5.097804879 podStartE2EDuration="6.752046633s" totalImagesPullingTime="1.654241754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:02.711697048 +0000 UTC m=+35.690389847" watchObservedRunningTime="2026-10-30 11:54:05.752046633 +0000 UTC m=+38.730739430" Oct 30 11:54:05.752361 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:05.752215 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g584t" podStartSLOduration=34.250760379 podStartE2EDuration="37.752207042s" totalImagesPullingTime="3.501446663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:05.750759957 +0000 UTC m=+38.729452765" watchObservedRunningTime="2026-10-30 11:54:05.752207042 +0000 UTC m=+38.730899843" Oct 30 11:54:05.758704 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:05.758672 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:05.865871 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:05.865802 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jz62m_535eda1f-d9e7-4908-b958-56f531b68fbb/dns-node-resolver/0.log" Oct 30 11:54:05.879093 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:05.879060 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:06.444533 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:06.444502 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:06.683220 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:06.683189 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pnvz9" event={"ID":"68aaec5d-0bc6-4e8b-a311-1797a055bb7b","Type":"ContainerStarted","Data":"ed98f088e333d22142e2d2f2629e99907d1d1786024b51c2572ba1cb1d0ded97"} Oct 30 11:54:06.729203 ip-10-0-138-23 kubenswrapper[2577]: I1030 11:54:06.729076 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pnvz9" podStartSLOduration=33.953517085 podStartE2EDuration="38.72905688s" totalImagesPullingTime="4.775539795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:06.727964953 +0000 UTC m=+39.706657751" watchObservedRunningTime="2026-10-30 11:54:06.72905688 +0000 UTC m=+39.707749678" Oct 30 11:54:06.812379 ip-10-0-138-23 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 11:54:06.820724 ip-10-0-138-23 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 11:54:06.820885 ip-10-0-138-23 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 11:54:06.820932 ip-10-0-138-23 systemd[1]: kubelet.service: Consumed 2.287s CPU time. Oct 30 11:54:06.836723 ip-10-0-138-23 systemd[1]: Starting Kubernetes Kubelet... Oct 30 11:54:06.935532 ip-10-0-138-23 kubenswrapper[4413]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:54:06.935532 ip-10-0-138-23 kubenswrapper[4413]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 11:54:06.935532 ip-10-0-138-23 kubenswrapper[4413]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:54:06.935532 ip-10-0-138-23 kubenswrapper[4413]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 11:54:06.937667 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937581 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:06.937667 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937667 4413 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937672 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937675 4413 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937678 4413 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937681 4413 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937684 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937687 4413 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937690 4413 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937692 4413 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937695 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937698 4413 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937701 4413 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937703 4413 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937706 4413 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937709 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937711 4413 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937713 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937716 4413 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937719 4413 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:06.937740 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937728 4413 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937731 4413 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937733 4413 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937736 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937739 4413 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937742 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937744 4413 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937747 4413 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937750 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937753 4413 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937756 4413 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937758 4413 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937761 4413 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937763 4413 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937766 4413 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937768 4413 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937771 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937773 4413 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937776 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937779 4413 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:06.938219 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937781 4413 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937785 4413 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937788 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937790 4413 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937793 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937795 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937798 4413 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937801 4413 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937803 4413 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937806 4413 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937808 4413 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937811 4413 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937813 4413 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937816 4413 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937819 4413 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937821 4413 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937824 4413 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937826 4413 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937829 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937832 4413 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:06.938731 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937835 4413 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937837 4413 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937839 4413 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937842 4413 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937844 4413 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937847 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937850 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937852 4413 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937855 4413 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937857 4413 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937860 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937863 4413 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937866 4413 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937868 4413 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937872 4413 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937877 4413 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937880 4413 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937883 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937888 4413 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:06.939400 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937892 4413 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937895 4413 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937898 4413 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937901 4413 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937904 4413 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937906 4413 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937909 4413 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937912 4413 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937914 4413 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937917 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937919 4413 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937922 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937925 4413 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937927 4413 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937930 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937932 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937934 4413 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937937 4413 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937940 4413 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:06.939880 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937945 4413 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937949 4413 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937952 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937957 4413 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937961 4413 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937965 4413 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937969 4413 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.937974 4413 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938740 4413 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938747 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938751 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938754 4413 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938757 4413 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938760 4413 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938763 4413 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938766 4413 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938769 4413 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938772 4413 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938774 4413 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:06.940372 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938777 4413 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938780 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938782 4413 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938785 4413 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938787 4413 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938790 4413 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938792 4413 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938795 4413 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938797 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938800 4413 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938803 4413 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938805 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938808 4413 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938810 4413 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938813 4413 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938815 4413 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938818 4413 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938822 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938825 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:06.940848 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938827 4413 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938830 4413 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938833 4413 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938836 4413 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938839 4413 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938841 4413 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938844 4413 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938846 4413 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938849 4413 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938851 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938854 4413 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938857 4413 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938859 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938862 4413 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938864 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938866 4413 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938869 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938872 4413 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938875 4413 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:06.941332 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938878 4413 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938881 4413 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938884 4413 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938886 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938889 4413 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938891 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938894 4413 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938896 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938899 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938902 4413 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938906 4413 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938909 4413 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938911 4413 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938914 4413 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938917 4413 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938919 4413 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938922 4413 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938925 4413 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938928 4413 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938946 4413 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:06.941820 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938950 4413 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938955 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938958 4413 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938961 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938964 4413 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938967 4413 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938970 4413 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938972 4413 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938975 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938977 4413 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938980 4413 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938982 4413 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938985 4413 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938987 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938990 4413 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938993 4413 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938995 4413 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.938998 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939000 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939003 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:06.942359 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939005 4413 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939008 4413 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939011 4413 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939016 4413 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939020 4413 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939023 4413 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939026 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939029 4413 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939033 4413 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939036 4413 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939038 4413 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939041 4413 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939044 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939046 4413 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939049 4413 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939051 4413 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939054 4413 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939550 4413 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939555 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:06.942840 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939559 4413 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939562 4413 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939564 4413 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939567 4413 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939570 4413 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939573 4413 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939575 4413 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939578 4413 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939580 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939583 4413 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939585 4413 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939588 4413 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939590 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939593 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939595 4413 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939598 4413 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939601 4413 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939604 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939607 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:06.943321 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939609 4413 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939612 4413 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939615 4413 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939618 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939621 4413 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939624 4413 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939626 4413 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939629 4413 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939631 4413 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939634 4413 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939636 4413 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939639 4413 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939641 4413 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939644 4413 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939646 4413 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939649 4413 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939652 4413 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939654 4413 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939668 4413 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939671 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:06.943814 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939675 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939678 4413 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939681 4413 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939683 4413 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939686 4413 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939688 4413 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939691 4413 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939693 4413 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939696 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939698 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939704 4413 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939707 4413 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939710 4413 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939713 4413 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939715 4413 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939719 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939722 4413 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939724 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939727 4413 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:06.944320 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939730 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939732 4413 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939736 4413 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939738 4413 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939741 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939743 4413 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939746 4413 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939748 4413 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939751 4413 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939754 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939756 4413 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939758 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939761 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939764 4413 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939767 4413 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939770 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939772 4413 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939775 4413 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939777 4413 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939780 4413 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:06.944783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939782 4413 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939785 4413 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939787 4413 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939794 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939796 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939799 4413 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939801 4413 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939804 4413 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939807 4413 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939809 4413 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939813 4413 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939816 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939818 4413 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939821 4413 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939823 4413 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939826 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939828 4413 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939831 4413 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939833 4413 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:06.945297 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939836 4413 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939840 4413 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939844 4413 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939850 4413 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939854 4413 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939857 4413 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.939860 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.939951 4413 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.939958 4413 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.939964 4413 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.939969 4413 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.939973 4413 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.939977 4413 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.939981 4413 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.939985 4413 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.939988 4413 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.939992 4413 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.939995 4413 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.939999 4413 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940002 4413 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940005 4413 flags.go:64] FLAG: --cgroup-root="" Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940008 4413 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 11:54:06.945810 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940011 4413 flags.go:64] FLAG: --client-ca-file="" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940014 4413 flags.go:64] FLAG: --cloud-provider="external" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940018 4413 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940025 4413 flags.go:64] FLAG: --cluster-domain="" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940028 4413 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940031 4413 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940035 4413 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940038 4413 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940042 4413 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940045 4413 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940048 4413 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940051 4413 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940055 4413 flags.go:64] FLAG: --contention-profiling="false" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940059 4413 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940063 4413 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940066 4413 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940069 4413 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940073 4413 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940076 4413 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940079 4413 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940082 4413 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940085 4413 flags.go:64] FLAG: --enable-server="true" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940087 4413 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940096 4413 flags.go:64] FLAG: --event-burst="100" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940099 4413 flags.go:64] FLAG: --event-qps="50" Oct 30 11:54:06.946704 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940102 4413 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940105 4413 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940108 4413 flags.go:64] FLAG: --eviction-hard="" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940113 4413 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940116 4413 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940119 4413 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940122 4413 flags.go:64] FLAG: --eviction-soft="" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940125 4413 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940128 4413 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940131 4413 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940135 4413 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940138 4413 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940141 4413 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940144 4413 flags.go:64] FLAG: --feature-gates="" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940148 4413 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940151 4413 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940154 4413 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940157 4413 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940160 4413 flags.go:64] FLAG: --healthz-port="10248" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940163 4413 flags.go:64] FLAG: --help="false" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940168 4413 flags.go:64] FLAG: --hostname-override="ip-10-0-138-23.ec2.internal" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940171 4413 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940174 4413 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 11:54:06.947432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940177 4413 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940180 4413 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940184 4413 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940187 4413 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940190 4413 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940192 4413 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940195 4413 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940198 4413 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940201 4413 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940204 4413 flags.go:64] FLAG: --kube-reserved="" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940207 4413 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940210 4413 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940213 4413 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940216 4413 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940219 4413 flags.go:64] FLAG: --lock-file="" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940222 4413 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940225 4413 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940228 4413 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940233 4413 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940236 4413 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940240 4413 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940243 4413 flags.go:64] FLAG: --logging-format="text" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940260 4413 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940264 4413 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 11:54:06.948089 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940267 4413 flags.go:64] FLAG: --manifest-url="" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940270 4413 flags.go:64] FLAG: --manifest-url-header="" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940274 4413 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940277 4413 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940281 4413 flags.go:64] FLAG: --max-pods="110" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940285 4413 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940288 4413 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940292 4413 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940297 4413 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940302 4413 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940306 4413 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940312 4413 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940322 4413 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940326 4413 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940328 4413 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940332 4413 flags.go:64] FLAG: --pod-cidr="" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940334 4413 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940337 4413 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940340 4413 flags.go:64] FLAG: --pods-per-core="0" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940343 4413 flags.go:64] FLAG: --port="10250" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940346 4413 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940348 4413 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a7d4e5c9786b6093" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940351 4413 flags.go:64] FLAG: --qos-reserved="" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940354 4413 flags.go:64] FLAG: --read-only-port="10255" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940357 4413 flags.go:64] FLAG: --register-node="true" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940360 4413 flags.go:64] FLAG: --register-with-taints="" Oct 30 11:54:06.948708 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940364 4413 flags.go:64] FLAG: --registry-burst="10" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940366 4413 flags.go:64] FLAG: --registry-qps="5" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940369 4413 flags.go:64] FLAG: --reserved-cpus="" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940373 4413 flags.go:64] FLAG: --reserved-memory="" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940377 4413 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940380 4413 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940383 4413 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940386 4413 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940389 4413 flags.go:64] FLAG: --runonce="false" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940393 4413 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940396 4413 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940400 4413 flags.go:64] FLAG: --seccomp-default="false" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940403 4413 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940406 4413 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940411 4413 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940414 4413 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940417 4413 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940420 4413 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940423 4413 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940426 4413 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940429 4413 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940432 4413 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940435 4413 flags.go:64] FLAG: --system-cgroups="" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940438 4413 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940443 4413 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 11:54:06.949354 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940446 4413 flags.go:64] FLAG: --tls-cert-file="" Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940449 4413 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940454 4413 flags.go:64] FLAG: --tls-min-version="" Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940457 4413 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940460 4413 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940463 4413 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940466 4413 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940468 4413 flags.go:64] FLAG: --v="2" Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940473 4413 flags.go:64] FLAG: --version="false" Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940477 4413 flags.go:64] FLAG: --vmodule="" Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940481 4413 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940485 4413 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940653 4413 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940657 4413 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940659 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940662 4413 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940665 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940667 4413 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940672 4413 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940675 4413 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940677 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940681 4413 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:06.950000 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940684 4413 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940687 4413 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940689 4413 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940692 4413 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940694 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940697 4413 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940700 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940703 4413 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940705 4413 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940708 4413 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940710 4413 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940713 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940715 4413 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940718 4413 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940720 4413 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940723 4413 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940726 4413 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940728 4413 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940731 4413 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:06.950562 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940733 4413 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940736 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940738 4413 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940742 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940744 4413 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940747 4413 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940749 4413 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940752 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940754 4413 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940758 4413 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940760 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940763 4413 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940766 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940769 4413 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940772 4413 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940774 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940777 4413 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940779 4413 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940782 4413 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940784 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:06.958618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940787 4413 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940790 4413 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940792 4413 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940795 4413 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940797 4413 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940801 4413 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940806 4413 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940809 4413 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940811 4413 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940814 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940816 4413 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940819 4413 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940821 4413 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940824 4413 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940827 4413 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940829 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940832 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940835 4413 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940838 4413 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940840 4413 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:06.959129 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940842 4413 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940846 4413 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940849 4413 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940852 4413 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940856 4413 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940859 4413 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940861 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940864 4413 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940867 4413 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940869 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940872 4413 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940874 4413 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940877 4413 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940879 4413 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940882 4413 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940884 4413 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940887 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940890 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940892 4413 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940895 4413 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:06.959641 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940897 4413 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:06.960137 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940900 4413 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:06.960137 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940902 4413 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:06.960137 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940905 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:06.960137 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940907 4413 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:06.960137 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940910 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:06.960137 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940913 4413 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:06.960137 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940917 4413 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:06.960137 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940920 4413 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:06.960137 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940926 4413 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:06.960137 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940929 4413 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:06.960137 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940931 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:06.960137 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940934 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:06.960137 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940938 4413 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:06.960137 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940941 4413 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:06.960137 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940943 4413 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:06.960137 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.940947 4413 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:06.960559 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.940973 4413 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:54:06.960559 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941023 4413 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:06.960559 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941027 4413 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:06.960559 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941030 4413 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:06.960559 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941034 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:06.960559 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941037 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:06.960559 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941039 4413 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:06.960559 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941042 4413 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:06.960559 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941044 4413 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:06.960559 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941047 4413 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:06.960559 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941050 4413 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:06.960559 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941052 4413 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:06.960559 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941055 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:06.960559 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941058 4413 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:06.960559 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941060 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:06.960559 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941062 4413 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941065 4413 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941068 4413 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941070 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941073 4413 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941075 4413 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941078 4413 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941081 4413 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941083 4413 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941086 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941092 4413 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941095 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941097 4413 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941104 4413 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941107 4413 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941109 4413 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941112 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941116 4413 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941118 4413 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:06.960948 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941121 4413 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941124 4413 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941126 4413 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941129 4413 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941131 4413 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941134 4413 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941136 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941154 4413 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941158 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941161 4413 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941164 4413 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941166 4413 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941169 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941172 4413 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941174 4413 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941177 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941180 4413 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941182 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941191 4413 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:06.961560 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941194 4413 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941197 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941199 4413 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941201 4413 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941204 4413 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941212 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941215 4413 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941218 4413 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941221 4413 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941224 4413 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941226 4413 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941231 4413 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941235 4413 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941239 4413 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941242 4413 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941259 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941263 4413 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941265 4413 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941268 4413 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941270 4413 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:06.962033 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941273 4413 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941275 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941278 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941281 4413 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941283 4413 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941286 4413 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941288 4413 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941291 4413 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941294 4413 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941296 4413 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941299 4413 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941301 4413 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941304 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941306 4413 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941308 4413 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941311 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941313 4413 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941316 4413 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941322 4413 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941328 4413 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:06.962531 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941330 4413 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941333 4413 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941336 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941340 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941342 4413 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941345 4413 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941348 4413 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941350 4413 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941353 4413 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941356 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941358 4413 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941361 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941365 4413 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.941369 4413 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.945162 4413 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.945177 4413 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.945195 4413 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.945201 4413 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945271 4413 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:06.963325 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945277 4413 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945280 4413 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945283 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945286 4413 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945288 4413 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945291 4413 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945294 4413 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945297 4413 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945299 4413 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945302 4413 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945305 4413 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945308 4413 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945310 4413 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945313 4413 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945316 4413 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945318 4413 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945321 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945323 4413 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945326 4413 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945328 4413 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:06.964007 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945331 4413 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945333 4413 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945336 4413 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945339 4413 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945341 4413 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945344 4413 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945347 4413 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945349 4413 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945353 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945356 4413 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945358 4413 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945361 4413 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945363 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945366 4413 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945369 4413 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945371 4413 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945374 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945376 4413 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945379 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:06.964729 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945381 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945384 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945387 4413 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945389 4413 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945392 4413 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945394 4413 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945397 4413 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945400 4413 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945402 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945406 4413 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945410 4413 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945413 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945417 4413 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945421 4413 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945424 4413 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945427 4413 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945429 4413 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945432 4413 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945435 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:06.965286 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945438 4413 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945440 4413 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945443 4413 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945446 4413 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945449 4413 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945452 4413 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945454 4413 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945457 4413 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945460 4413 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945462 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945465 4413 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945468 4413 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945470 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945473 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945475 4413 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945478 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945480 4413 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945483 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945486 4413 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:06.966111 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945488 4413 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945490 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945493 4413 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945495 4413 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945498 4413 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945500 4413 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945503 4413 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945505 4413 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945508 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945510 4413 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945513 4413 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945516 4413 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945518 4413 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945521 4413 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945524 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945527 4413 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945529 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945532 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945535 4413 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:06.967023 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945537 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:06.967930 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945540 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:06.967930 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945542 4413 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:06.967930 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945545 4413 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:06.967930 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945548 4413 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:06.967930 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945551 4413 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:06.967930 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945553 4413 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:06.967930 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945556 4413 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:06.967930 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.945558 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:06.967930 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.945594 4413 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:54:06.967930 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946025 4413 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:06.967930 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946331 4413 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:06.967930 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946337 4413 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:06.967930 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946341 4413 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:06.967930 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946346 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:06.967930 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946351 4413 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:06.967930 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946356 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946367 4413 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946371 4413 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946376 4413 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946380 4413 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946384 4413 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946388 4413 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946392 4413 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946396 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946400 4413 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946405 4413 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946409 4413 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946413 4413 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946419 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946428 4413 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946432 4413 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946437 4413 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946442 4413 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946447 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946452 4413 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:06.968638 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946456 4413 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946460 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946464 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946469 4413 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946473 4413 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946478 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946488 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946493 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946498 4413 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946502 4413 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946507 4413 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946511 4413 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946515 4413 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946519 4413 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946523 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946527 4413 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946531 4413 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946535 4413 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946540 4413 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946549 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:06.969487 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946553 4413 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946558 4413 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946562 4413 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946567 4413 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946571 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946576 4413 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946580 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946584 4413 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946588 4413 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946592 4413 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946597 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946606 4413 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946611 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946615 4413 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946620 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946624 4413 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946628 4413 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946632 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946636 4413 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:06.970343 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946640 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:06.971165 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946645 4413 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:06.971165 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946650 4413 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:06.971165 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946654 4413 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:06.971165 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946658 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:06.971165 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946667 4413 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:06.971165 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946671 4413 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:06.971165 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946675 4413 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:06.971165 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946679 4413 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:06.971165 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946684 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:06.971165 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946688 4413 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:06.971165 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946693 4413 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:06.971165 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946699 4413 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:06.971165 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946705 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:06.971165 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946711 4413 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:06.971165 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946719 4413 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:06.971165 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946725 4413 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:06.971165 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946734 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:06.971165 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946740 4413 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946744 4413 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946748 4413 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946753 4413 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946757 4413 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946761 4413 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946765 4413 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946769 4413 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946773 4413 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946777 4413 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946781 4413 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946785 4413 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946794 4413 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946798 4413 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946802 4413 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946806 4413 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946811 4413 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946815 4413 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946819 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946823 4413 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:06.971986 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946827 4413 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:06.972727 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946832 4413 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:06.972727 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.946836 4413 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:06.972727 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.946955 4413 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 11:54:06.972727 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947188 4413 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 11:54:06.972727 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947196 4413 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 11:54:06.972727 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947201 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 11:54:06.972727 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947206 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 11:54:06.972727 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947211 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 11:54:06.972727 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947214 4413 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 11:54:06.972727 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947219 4413 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 11:54:06.972727 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947223 4413 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 11:54:06.972727 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947239 4413 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 11:54:06.972727 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947243 4413 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 11:54:06.972727 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947267 4413 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 11:54:06.972727 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947271 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 11:54:06.972727 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947275 4413 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947279 4413 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947283 4413 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947287 4413 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947292 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947296 4413 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947300 4413 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947304 4413 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947308 4413 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947312 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947316 4413 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947320 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947324 4413 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947329 4413 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947333 4413 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947338 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947342 4413 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947346 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947350 4413 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947355 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 11:54:06.973447 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947359 4413 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 11:54:06.975338 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947363 4413 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 11:54:06.975338 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947367 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 11:54:06.975338 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947371 4413 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 11:54:06.975338 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947376 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 11:54:06.975338 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947380 4413 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 11:54:06.975338 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947384 4413 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 11:54:06.975338 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947387 4413 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 11:54:06.975338 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947391 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 11:54:06.975338 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947396 4413 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 11:54:06.975338 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947400 4413 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 11:54:06.975338 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947403 4413 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 11:54:06.975338 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947406 4413 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 11:54:06.975338 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947409 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 11:54:06.975338 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947411 4413 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 11:54:06.975338 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947414 4413 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 11:54:06.975338 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947418 4413 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 11:54:06.975338 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947422 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 11:54:06.975338 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947426 4413 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 11:54:06.973968 ip-10-0-138-23 systemd[1]: Started Kubernetes Kubelet. Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947430 4413 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947434 4413 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947437 4413 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947440 4413 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947443 4413 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947446 4413 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947449 4413 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947452 4413 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947455 4413 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947458 4413 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947460 4413 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947463 4413 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947466 4413 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947470 4413 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947473 4413 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947475 4413 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947478 4413 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947481 4413 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947484 4413 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 11:54:06.976422 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947486 4413 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947489 4413 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947493 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947496 4413 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947498 4413 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947501 4413 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947504 4413 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947506 4413 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947508 4413 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947511 4413 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947513 4413 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947516 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947518 4413 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947521 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947523 4413 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947526 4413 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947528 4413 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947531 4413 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947533 4413 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947536 4413 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 11:54:06.978011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947539 4413 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947541 4413 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947544 4413 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947546 4413 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947549 4413 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947552 4413 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947556 4413 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947558 4413 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947561 4413 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947563 4413 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947566 4413 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947569 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947573 4413 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947575 4413 feature_gate.go:386] unrecognized feature gate: Example Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947578 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:06.947581 4413 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.947644 4413 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.948654 4413 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.948718 4413 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 11:54:06.978889 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.948900 4413 server.go:1014] "Starting client certificate rotation" Oct 30 11:54:06.979792 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.949035 4413 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 11:54:06.979792 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.949079 4413 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 11:48:28 +0000 UTC" deadline="2028-06-21 23:45:43.059114254 +0000 UTC" Oct 30 11:54:06.979792 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.949116 4413 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14411h51m36.110000793s" Oct 30 11:54:06.979792 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.949395 4413 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 11:54:06.979792 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.950520 4413 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 11:54:06.979792 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.952032 4413 log.go:25] "Validated CRI v1 runtime API" Oct 30 11:54:06.979792 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.953574 4413 log.go:25] "Validated CRI v1 image API" Oct 30 11:54:06.979792 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.954316 4413 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 11:54:06.979792 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.955738 4413 fs.go:135] Filesystem UUIDs: map[39daed18-86e6-4776-868c-7cd05318025c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f1f48a21-c751-4b0b-870d-d1d7bc18057b:/dev/nvme0n1p3] Oct 30 11:54:06.980121 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.955758 4413 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/052f9555db593197a01666688c3a2781a42079e966f27299450d6fbc934d7f33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/052f9555db593197a01666688c3a2781a42079e966f27299450d6fbc934d7f33/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1345f65fd1b97393ec76019253855fefed59956dae9f3758ee3f67a0a1301d4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1345f65fd1b97393ec76019253855fefed59956dae9f3758ee3f67a0a1301d4c/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17b883f17b8183ee1f8d45b16f43ee7d69bd5f0cf23fac2fbf8fff150588df71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17b883f17b8183ee1f8d45b16f43ee7d69bd5f0cf23fac2fbf8fff150588df71/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ac4c9518583c4ba4571dd43309a5be645d0edb1cc1a0e0e93bc0fe8c7a90df7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ac4c9518583c4ba4571dd43309a5be645d0edb1cc1a0e0e93bc0fe8c7a90df7/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/376ea76ae9c666fdffe54b57c639f33b6068a1b0f6df2c719aa300065016161e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/376ea76ae9c666fdffe54b57c639f33b6068a1b0f6df2c719aa300065016161e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b39fd27f30cf3b4ecfe2293c4990fce79257e8dbe5e18cdcb6d41af03f02b92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b39fd27f30cf3b4ecfe2293c4990fce79257e8dbe5e18cdcb6d41af03f02b92/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e77dbd0b1b7ea3f480213918525725419f670a27da07410f16371b00928f042/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e77dbd0b1b7ea3f480213918525725419f670a27da07410f16371b00928f042/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81192aec7523c3e2ed796b94d26f534c588906851cbf0bef6f3279b0761b7bf2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81192aec7523c3e2ed796b94d26f534c588906851cbf0bef6f3279b0761b7bf2/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84dd1f111347bc3a986759810ffd4928000cb6c86746b5216ca22a74cdb5bf83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84dd1f111347bc3a986759810ffd4928000cb6c86746b5216ca22a74cdb5bf83/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a090e0699c8024b05ff126f7ca48701b8dc97108a9f5364ed1d6af4a1598521e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a090e0699c8024b05ff126f7ca48701b8dc97108a9f5364ed1d6af4a1598521e/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c69d9dfb491728afa0d6a3d6a46b2802ddfd18ee95dd01558ea41a8dc1ed1ff3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c69d9dfb491728afa0d6a3d6a46b2802ddfd18ee95dd01558ea41a8dc1ed1ff3/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6ad89aef93286ff606e8576d07abe2ef749e69c10bcc5d88c91e1e9e65c000e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6ad89aef93286ff606e8576d07abe2ef749e69c10bcc5d88c91e1e9e65c000e/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec4ae230280ca3268fe9fc538f84aae697eea73bf14c363e17a60963812925ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec4ae230280ca3268fe9fc538f84aae697eea73bf14c363e17a60963812925ad/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcdbe60b8e623618932dfe168463e773fa7f2ede0126139390f7f77cfeefaaf5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcdbe60b8e623618932dfe168463e773fa7f2ede0126139390f7f77cfeefaaf5/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/125e1073-73c6-4aae-9c1b-4dc6e047e241/volumes/kubernetes.io~projected/kube-api-access-h5wc8:{mountpoint:/var/lib/kubelet/pods/125e1073-73c6-4aae-9c1b-4dc6e047e241/volumes/kubernetes.io~projected/kube-api-access-h5wc8 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/125e1073-73c6-4aae-9c1b-4dc6e047e241/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/125e1073-73c6-4aae-9c1b-4dc6e047e241/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a3afa91-aff6-455b-aeda-7275cd8ac8e0/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1a3afa91-aff6-455b-aeda-7275cd8ac8e0/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c3ac2bd-4472-4779-8ca5-460160cdedbb/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1c3ac2bd-4472-4779-8ca5-460160cdedbb/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c3ac2bd-4472-4779-8ca5-460160cdedbb/volumes/kubernetes.io~projected/kube-api-access-brmjx:{mountpoint:/var/lib/kubelet/pods/1c3ac2bd-4472-4779-8ca5-460160cdedbb/volumes/kubernetes.io~projected/kube-api-access-brmjx major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c3ac2bd-4472-4779-8ca5-460160cdedbb/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1c3ac2bd-4472-4779-8ca5-460160cdedbb/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27f694f1-e1de-483f-a5c5-250f52dae768/volumes/kubernetes.io~projected/kube-api-access-bw9h2:{mountpoint:/var/lib/kubelet/pods/27f694f1-e1de-483f-a5c5-250f52dae768/volumes/kubernetes.io~projected/kube-api-access-bw9h2 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39ef6d56-2baa-4f93-9268-12a74e0ef903/volumes/kubernetes.io~projected/kube-api-access-fhwsn:{mountpoint:/var/lib/kubelet/pods/39ef6d56-2baa-4f93-9268-12a74e0ef903/volumes/kubernetes.io~projected/kube-api-access-fhwsn major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd/volumes/kubernetes.io~projected/kube-api-access-tz6mk:{mountpoint:/var/lib/kubelet/pods/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd/volumes/kubernetes.io~projected/kube-api-access-tz6mk major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52ede1dc-817e-434c-874a-2db2a9c10f52/volumes/kubernetes.io~projected/kube-api-access-kbv7k:{mountpoint:/var/lib/kubelet/pods/52ede1dc-817e-434c-874a-2db2a9c10f52/volumes/kubernetes.io~projected/kube-api-access-kbv7k major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/535eda1f-d9e7-4908-b958-56f531b68fbb/volumes/kubernetes.io~projected/kube-api-access-2vkxp:{mountpoint:/var/lib/kubelet/pods/535eda1f-d9e7-4908-b958-56f531b68fbb/volumes/kubernetes.io~projected/kube-api-access-2vkxp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390/volumes/kubernetes.io~projected/kube-api-access-m2vgd:{mountpoint:/var/lib/kubelet/pods/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390/volumes/kubernetes.io~projected/kube-api-access-m2vgd major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68aaec5d-0bc6-4e8b-a311-1797a055bb7b/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/68aaec5d-0bc6-4e8b-a311-1797a055bb7b/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82e6695b-1b88-4372-9d00-ddf341253c59/volumes/kubernetes.io~projected/kube-api-access-kg275:{mountpoint:/var/lib/kubelet/pods/82e6695b-1b88-4372-9d00-ddf341253c59/volumes/kubernetes.io~projected/kube-api-access-kg275 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/864715b0-07ec-483a-8401-089c747422cb/volumes/kubernetes.io~projected/kube-api-access-w7dwl:{mountpoint:/var/lib/kubelet/pods/864715b0-07ec-483a-8401-089c747422cb/volumes/kubernetes.io~projected/kube-api-access-w7dwl major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b4fd8cb-a92d-40b2-8133-b04c126332e7/volumes/kubernetes.io~projected/kube-api-access-9nqkz:{mountpoint:/var/lib/kubelet/pods/8b4fd8cb-a92d-40b2-8133-b04c126332e7/volumes/kubernetes.io~projected/kube-api-access-9nqkz major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2b6ec57-b883-408a-ab01-abd192572510/volumes/kubernetes.io~projected/kube-api-access-vx4gc:{mountpoint:/var/lib/kubelet/pods/e2b6ec57-b883-408a-ab01-abd192572510/volumes/kubernetes.io~projected/kube-api-access-vx4gc major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edbd4ad4-8205-460e-9385-3151a16e7b68/volumes/kubernetes.io~projected/kube-api-access-hhm5c:{mountpoint:/var/lib/kubelet/pods/edbd4ad4-8205-460e-9385-3151a16e7b68/volumes/kubernetes.io~projected/kube-api-access-hhm5c major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc833d39-b781-486b-9fcc-549cba555ed9/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/fc833d39-b781-486b-9fcc-549cba555ed9/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc833d39-b781-486b-9fcc-549cba555ed9/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/fc833d39-b781-486b-9fcc-549cba555ed9/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc833d39-b781-486b-9fcc-549cba555ed9/volumes/kubernetes.io~projected/kube-api-access-hzcmw:{mountpoint:/var/lib/kubelet/pods/fc833d39-b781-486b-9fcc-549cba555ed9/volumes/kubernetes.io~projected/kube-api-access-hzcmw major:0 minor:67 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/1e673431670f2f55f34a72ebcd40fce0bcbd0150fc4698b4780043400cb13e91/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/4d750973aea4b72f05df0994cd417adb81df156068cfebbf8a2cdec8c47df76b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/93a8d56437a26d63e62338d27019baa5a65b70250e2a914ee2861c94e1c4dbdf/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/f23ec344bfe614bc22a7292fc379c4090cc0281972a84c86a759d388d8fc8a8a/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d4af2562259c7489bea3bd653d7646afff06b1a6d95d50d7ff515f1c432e7515/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3172a0bd4dd7c7712bb2eb33862d061e16801fb1d344f3f3031ce3b6db554f8f/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a782d503f86492bac89f7ad04680b8495c14fc7fe6aabbf859fc037b9c47bdfc/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8021acb5d656fc6e779e742e6b0f7cab30f557348f8870320abd34776f825712/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/8888b206640c22a30cc7757d7b496b136dffe36a4976e81a103d2fb3826e7313/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/8ee8584c2b516f37ce97f8ea740c72b45358c343eeca5ba5ce517640beb3ee4e/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/9fbad56ec1862a4263ec12136f783f26059eb7dff65c3bbb80c3ce38dbdfdfa0/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/d5e06a7092e8c01669d5ce790fbe6ff6d7e332a0c78d787e19ae9d3c8802d3cd/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/6a12eef27f50b87a52a49765c04ca39fe5e13718e845ed59a5884a0b99d2d19a/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/73067921b690ba0eeb601c5e6e012d7ea998c3552c04e6e727c815a6e8833648/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ea98ffb8f27f23bbf50e8d3f2c0051a32dc4f94340639a03093f7789bf8f3b0a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/83d54e46c2a348b7947e81a6f09dcc86afe8bd15e7557529c3cfea9580443271/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a9471fd2e9dd827c0f4eacbbd1b5b463392f6e39a0b2e1be62e2ba15da305b39/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d4af09a1ee96e57524deaf89a0b1b6b189e45be48dd0f1f81e0e446800cdf1bc/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/9f81ae52fbc52d26f44ebf3217cce8f0b641a162242dce0a322e835142d70d64/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/6c43c4faaf5739108f876cb3456efe2572e62c86a7308e49feaf53548358663c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/4fc349066b54e3253925f8843846af2d2a128a1a79085c0a038c02b60ce9daa4/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/570fb019515e6e28a692128df7e8b5db29662181d034b607488a75e1e34ae44e/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/815dfb8d7cf6ac48a3f5c10075b21b57e5596420f4c3e7d342458d8eaaff286e/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/f4122dc9db900c17d62a1c54ea551c8f1a1f162fc02a092dd13d6ec6c2a31a62/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/1ffea3066d5c2501807f01cc4b0cc02cc161626462cba7dc19dacf574cb47a84/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/6de321b59306bf6e079856780250eadc493c245ee464615655e222cb19b3a452/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/8834fe0bb8567de51452e42a786495c4ac3922361fe3f1544cb44e6aea5057a1/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/b8de4f0e6ff05208031cc978f5911aeaad0a957c9ea2210b2a071e25e5fb8b93/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/3eacd119f0c1315d1aa0268ffb0b9e0a0ca24df4b8c85d738cf3db77876ebbc1/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/aec7e43599cf78e12ff6f4ef4d0fd466bf35d770524db5d36a48248e7610863e/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/956f46db01c9d865ad3ff4122d9619d6e09ab5e1e894cb2ee2dc61eb2bc3687f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/973a327dc78fa9c272fddb8bbd3be4f0b1d9f0761c60c2776675081d5bddacfd/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/9105bb2ca4809a51ce5a792b82dd42474cccb85750e3bc8978c38647d10a60d9/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/a3e6ee693e73ca5d0f82897ad0419e317bc5bf5a3dce4309f9ed0c650ea41933/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/e0fa7f21d5e3554f0a768492e8f4aeb6f8d0c30f723e794a01848ef30ba737d4/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/54791120a004f8e5c9a2b90f34a06ef1f15762d68ac2afc989eb67246cfdbc22/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/5d7054992a9ef65315e838a429b4e1ea5de0fe0977ab62b9eb2afea53227a2c3/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-95:{mountpoint:/var/lib/containers/storage/overlay/e3a5b60cafd81c0a313b3a351c0cdb49929bd0ae3930422d664d9683743f9972/merged major:0 minor:95 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/a2eb3a36f8250d4bef32b784f8b0dd32b66c1c4c723716c7278028db2b50a9ca/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.961943 4413 manager.go:217] Machine: {Timestamp:2026-10-30 11:54:06.961478429 +0000 UTC m=+0.071321302 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099865 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28c64b7420621ac5361e53ab0550a8 SystemUUID:ec28c64b-7420-621a-c536-1e53ab0550a8 BootID:1ddfa322-0634-48ae-b3e0-a31d7cad26d0 Filesystems:[{Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-95 DeviceMajor:0 DeviceMinor:95 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e2b6ec57-b883-408a-ab01-abd192572510/volumes/kubernetes.io~projected/kube-api-access-vx4gc DeviceMajor:0 DeviceMinor:60 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c3ac2bd-4472-4779-8ca5-460160cdedbb/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/125e1073-73c6-4aae-9c1b-4dc6e047e241/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/68aaec5d-0bc6-4e8b-a311-1797a055bb7b/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:214 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fc833d39-b781-486b-9fcc-549cba555ed9/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd/volumes/kubernetes.io~projected/kube-api-access-tz6mk DeviceMajor:0 DeviceMinor:61 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/535eda1f-d9e7-4908-b958-56f531b68fbb/volumes/kubernetes.io~projected/kube-api-access-2vkxp DeviceMajor:0 DeviceMinor:63 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/27f694f1-e1de-483f-a5c5-250f52dae768/volumes/kubernetes.io~projected/kube-api-access-bw9h2 DeviceMajor:0 DeviceMinor:65 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b4fd8cb-a92d-40b2-8133-b04c126332e7/volumes/kubernetes.io~projected/kube-api-access-9nqkz DeviceMajor:0 DeviceMinor:215 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82e6695b-1b88-4372-9d00-ddf341253c59/volumes/kubernetes.io~projected/kube-api-access-kg275 DeviceMajor:0 DeviceMinor:209 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1c3ac2bd-4472-4779-8ca5-460160cdedbb/volumes/kubernetes.io~projected/kube-api-access-brmjx DeviceMajor:0 DeviceMinor:66 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e77dbd0b1b7ea3f480213918525725419f670a27da07410f16371b00928f042/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/125e1073-73c6-4aae-9c1b-4dc6e047e241/volumes/kubernetes.io~projected/kube-api-access-h5wc8 DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc833d39-b781-486b-9fcc-549cba555ed9/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/052f9555db593197a01666688c3a2781a42079e966f27299450d6fbc934d7f33/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6ad89aef93286ff606e8576d07abe2ef749e69c10bcc5d88c91e1e9e65c000e/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a090e0699c8024b05ff126f7ca48701b8dc97108a9f5364ed1d6af4a1598521e/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1c3ac2bd-4472-4779-8ca5-460160cdedbb/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39ef6d56-2baa-4f93-9268-12a74e0ef903/volumes/kubernetes.io~projected/kube-api-access-fhwsn DeviceMajor:0 DeviceMinor:208 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84dd1f111347bc3a986759810ffd4928000cb6c86746b5216ca22a74cdb5bf83/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1a3afa91-aff6-455b-aeda-7275cd8ac8e0/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/52ede1dc-817e-434c-874a-2db2a9c10f52/volumes/kubernetes.io~projected/kube-api-access-kbv7k DeviceMajor:0 DeviceMinor:64 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c69d9dfb491728afa0d6a3d6a46b2802ddfd18ee95dd01558ea41a8dc1ed1ff3/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17b883f17b8183ee1f8d45b16f43ee7d69bd5f0cf23fac2fbf8fff150588df71/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390/volumes/kubernetes.io~projected/kube-api-access-m2vgd DeviceMajor:0 DeviceMinor:201 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/376ea76ae9c666fdffe54b57c639f33b6068a1b0f6df2c719aa300065016161e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/edbd4ad4-8205-460e-9385-3151a16e7b68/volumes/kubernetes.io~projected/kube-api-access-hhm5c DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81192aec7523c3e2ed796b94d26f534c588906851cbf0bef6f3279b0761b7bf2/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1345f65fd1b97393ec76019253855fefed59956dae9f3758ee3f67a0a1301d4c/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b39fd27f30cf3b4ecfe2293c4990fce79257e8dbe5e18cdcb6d41af03f02b92/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec4ae230280ca3268fe9fc538f84aae697eea73bf14c363e17a60963812925ad/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc833d39-b781-486b-9fcc-549cba555ed9/volumes/kubernetes.io~projected/kube-api-access-hzcmw DeviceMajor:0 DeviceMinor:67 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ac4c9518583c4ba4571dd43309a5be645d0edb1cc1a0e0e93bc0fe8c7a90df7/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/864715b0-07ec-483a-8401-089c747422cb/volumes/kubernetes.io~projected/kube-api-access-w7dwl DeviceMajor:0 DeviceMinor:62 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcdbe60b8e623618932dfe168463e773fa7f2ede0126139390f7f77cfeefaaf5/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1345f65fd1b9739 MacAddress:92:5c:4f:95:84:f0 Speed:10000 Mtu:8901} {Name:84dd1f111347bc3 MacAddress:6e:e2:d1:2f:7b:99 Speed:10000 Mtu:8901} {Name:a090e0699c8024b MacAddress:aa:06:82:df:7a:67 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e9:0a:87:93 Speed:0 Mtu:9001} {Name:br-int MacAddress:8a:19:e8:7c:e4:ea Speed:0 Mtu:8901} {Name:c69d9dfb491728a MacAddress:7e:59:93:58:8b:bf Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e9:0a:87:93 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:da:ca:57:1c:8c:55 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:6a:4f:11:f1:08:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.962183 4413 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.962225 4413 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.962775 4413 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.962843 4413 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-23.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.963155 4413 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.963167 4413 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.963186 4413 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.963204 4413 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.963349 4413 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.963613 4413 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.963789 4413 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.964097 4413 kubelet.go:493] "Attempting to sync node with API server" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.964141 4413 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.964156 4413 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.964166 4413 kubelet.go:405] "Adding apiserver pod source" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.964175 4413 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.964920 4413 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.964931 4413 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.965437 4413 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968170 4413 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968182 4413 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968329 4413 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968597 4413 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968640 4413 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968759 4413 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968777 4413 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968790 4413 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968799 4413 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968809 4413 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968818 4413 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968827 4413 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968837 4413 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968848 4413 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968857 4413 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968872 4413 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968885 4413 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968922 4413 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.968932 4413 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.973400 4413 server.go:1263] "Started kubelet" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.973478 4413 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.973503 4413 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.974056 4413 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.974388 4413 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.976690 4413 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.977004 4413 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.977122 4413 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 11:48:30 +0000 UTC" deadline="2028-05-18 16:20:04.727911423 +0000 UTC" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.977160 4413 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13588h25m57.750754758s" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.977209 4413 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.977219 4413 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.977308 4413 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:06.977389 4413 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-23.ec2.internal\" not found" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.978914 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39ef6d56-2baa-4f93-9268-12a74e0ef903" volumeName="kubernetes.io/projected/39ef6d56-2baa-4f93-9268-12a74e0ef903-kube-api-access-fhwsn" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.978943 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="535eda1f-d9e7-4908-b958-56f531b68fbb" volumeName="kubernetes.io/empty-dir/535eda1f-d9e7-4908-b958-56f531b68fbb-tmp-dir" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.978958 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c3ac2bd-4472-4779-8ca5-460160cdedbb" volumeName="kubernetes.io/secret/1c3ac2bd-4472-4779-8ca5-460160cdedbb-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.978971 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc833d39-b781-486b-9fcc-549cba555ed9" volumeName="kubernetes.io/empty-dir/fc833d39-b781-486b-9fcc-549cba555ed9-tmp" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.978984 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c3ac2bd-4472-4779-8ca5-460160cdedbb" volumeName="kubernetes.io/configmap/1c3ac2bd-4472-4779-8ca5-460160cdedbb-ovnkube-config" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.978996 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c3ac2bd-4472-4779-8ca5-460160cdedbb" volumeName="kubernetes.io/configmap/1c3ac2bd-4472-4779-8ca5-460160cdedbb-env-overrides" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979010 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62adf1a5-f87e-4cbc-86e5-bcd2cefbd390" volumeName="kubernetes.io/projected/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-kube-api-access-m2vgd" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979023 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27f694f1-e1de-483f-a5c5-250f52dae768" volumeName="kubernetes.io/projected/27f694f1-e1de-483f-a5c5-250f52dae768-kube-api-access-bw9h2" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979039 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c3ac2bd-4472-4779-8ca5-460160cdedbb" volumeName="kubernetes.io/configmap/1c3ac2bd-4472-4779-8ca5-460160cdedbb-ovnkube-script-lib" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979051 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="864715b0-07ec-483a-8401-089c747422cb" volumeName="kubernetes.io/configmap/864715b0-07ec-483a-8401-089c747422cb-cni-binary-copy" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979063 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edbd4ad4-8205-460e-9385-3151a16e7b68" volumeName="kubernetes.io/configmap/edbd4ad4-8205-460e-9385-3151a16e7b68-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979074 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edbd4ad4-8205-460e-9385-3151a16e7b68" volumeName="kubernetes.io/projected/edbd4ad4-8205-460e-9385-3151a16e7b68-kube-api-access-hhm5c" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979086 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc833d39-b781-486b-9fcc-549cba555ed9" volumeName="kubernetes.io/projected/fc833d39-b781-486b-9fcc-549cba555ed9-kube-api-access-hzcmw" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979099 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c3ac2bd-4472-4779-8ca5-460160cdedbb" volumeName="kubernetes.io/projected/1c3ac2bd-4472-4779-8ca5-460160cdedbb-kube-api-access-brmjx" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979144 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27f694f1-e1de-483f-a5c5-250f52dae768" volumeName="kubernetes.io/configmap/27f694f1-e1de-483f-a5c5-250f52dae768-serviceca" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979156 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52ede1dc-817e-434c-874a-2db2a9c10f52" volumeName="kubernetes.io/projected/52ede1dc-817e-434c-874a-2db2a9c10f52-kube-api-access-kbv7k" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979168 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="535eda1f-d9e7-4908-b958-56f531b68fbb" volumeName="kubernetes.io/projected/535eda1f-d9e7-4908-b958-56f531b68fbb-kube-api-access-2vkxp" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979181 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62adf1a5-f87e-4cbc-86e5-bcd2cefbd390" volumeName="kubernetes.io/configmap/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-config-volume" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979193 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="864715b0-07ec-483a-8401-089c747422cb" volumeName="kubernetes.io/configmap/864715b0-07ec-483a-8401-089c747422cb-multus-daemon-config" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979204 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2b6ec57-b883-408a-ab01-abd192572510" volumeName="kubernetes.io/projected/e2b6ec57-b883-408a-ab01-abd192572510-kube-api-access-vx4gc" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979218 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edbd4ad4-8205-460e-9385-3151a16e7b68" volumeName="kubernetes.io/configmap/edbd4ad4-8205-460e-9385-3151a16e7b68-cni-binary-copy" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979230 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="125e1073-73c6-4aae-9c1b-4dc6e047e241" volumeName="kubernetes.io/projected/125e1073-73c6-4aae-9c1b-4dc6e047e241-kube-api-access-h5wc8" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979261 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a3afa91-aff6-455b-aeda-7275cd8ac8e0" volumeName="kubernetes.io/configmap/1a3afa91-aff6-455b-aeda-7275cd8ac8e0-konnectivity-ca" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979276 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a3afa91-aff6-455b-aeda-7275cd8ac8e0" volumeName="kubernetes.io/secret/1a3afa91-aff6-455b-aeda-7275cd8ac8e0-agent-certs" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979287 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62adf1a5-f87e-4cbc-86e5-bcd2cefbd390" volumeName="kubernetes.io/empty-dir/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-tmp-dir" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979299 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b4fd8cb-a92d-40b2-8133-b04c126332e7" volumeName="kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979311 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edbd4ad4-8205-460e-9385-3151a16e7b68" volumeName="kubernetes.io/configmap/edbd4ad4-8205-460e-9385-3151a16e7b68-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 11:54:06.980274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979324 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc833d39-b781-486b-9fcc-549cba555ed9" volumeName="kubernetes.io/empty-dir/fc833d39-b781-486b-9fcc-549cba555ed9-etc-tuned" seLinuxMountContext="" Oct 30 11:54:06.982785 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979341 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd" volumeName="kubernetes.io/projected/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-kube-api-access-tz6mk" seLinuxMountContext="" Oct 30 11:54:06.982785 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979364 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68aaec5d-0bc6-4e8b-a311-1797a055bb7b" volumeName="kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret" seLinuxMountContext="" Oct 30 11:54:06.982785 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979378 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82e6695b-1b88-4372-9d00-ddf341253c59" volumeName="kubernetes.io/empty-dir/82e6695b-1b88-4372-9d00-ddf341253c59-data-volume" seLinuxMountContext="" Oct 30 11:54:06.982785 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979389 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82e6695b-1b88-4372-9d00-ddf341253c59" volumeName="kubernetes.io/projected/82e6695b-1b88-4372-9d00-ddf341253c59-kube-api-access-kg275" seLinuxMountContext="" Oct 30 11:54:06.982785 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979401 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="864715b0-07ec-483a-8401-089c747422cb" volumeName="kubernetes.io/projected/864715b0-07ec-483a-8401-089c747422cb-kube-api-access-w7dwl" seLinuxMountContext="" Oct 30 11:54:06.982785 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979412 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52ede1dc-817e-434c-874a-2db2a9c10f52" volumeName="kubernetes.io/configmap/52ede1dc-817e-434c-874a-2db2a9c10f52-iptables-alerter-script" seLinuxMountContext="" Oct 30 11:54:06.982785 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979424 4413 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="125e1073-73c6-4aae-9c1b-4dc6e047e241" volumeName="kubernetes.io/secret/125e1073-73c6-4aae-9c1b-4dc6e047e241-cert" seLinuxMountContext="" Oct 30 11:54:06.982785 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979436 4413 reconstruct.go:97] "Volume reconstruction finished" Oct 30 11:54:06.982785 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.979445 4413 reconciler.go:29] "Reconciler: start to sync state" Oct 30 11:54:06.985539 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.983791 4413 factory.go:55] Registering systemd factory Oct 30 11:54:06.985539 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.983812 4413 factory.go:223] Registration of the systemd container factory successfully Oct 30 11:54:06.985539 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.983830 4413 server.go:317] "Adding debug handlers to kubelet server" Oct 30 11:54:06.993575 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.993437 4413 factory.go:153] Registering CRI-O factory Oct 30 11:54:06.993805 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.993792 4413 factory.go:223] Registration of the crio container factory successfully Oct 30 11:54:06.993899 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.993879 4413 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 11:54:06.993953 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.993902 4413 factory.go:103] Registering Raw factory Oct 30 11:54:06.993953 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.993916 4413 manager.go:1199] Started watching for new ooms in manager Oct 30 11:54:06.994307 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:06.994290 4413 manager.go:319] Starting recovery of all containers Oct 30 11:54:07.009584 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.009529 4413 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 11:54:07.011275 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.011236 4413 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 11:54:07.011275 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.011276 4413 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 11:54:07.011419 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.011329 4413 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 11:54:07.011419 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:07.011387 4413 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 11:54:07.021689 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.021662 4413 generic.go:356] "Generic (PLEG): container finished" podID="3e9967fa0ac01d9b4f8c399a65ef8efb" containerID="f8e5a280c6002d9ccb9d53c330f406f1fe5f8714903135b2ef2f4e517d5f9b4e" exitCode=0 Oct 30 11:54:07.026044 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.026026 4413 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 11:54:07.027216 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.027189 4413 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:54:07.027587 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.027566 4413 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:54:07.031928 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.031905 4413 generic.go:356] "Generic (PLEG): container finished" podID="edbd4ad4-8205-460e-9385-3151a16e7b68" containerID="5bbe5e23362fdd4cc5bea195b335b40c011ddc779695356a74e4f31dfee003c2" exitCode=0 Oct 30 11:54:07.031928 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.031929 4413 generic.go:356] "Generic (PLEG): container finished" podID="edbd4ad4-8205-460e-9385-3151a16e7b68" containerID="0bb8663f233466d320137adf037d81661cc1c452909e3fc4a7d04bbd581c1db5" exitCode=0 Oct 30 11:54:07.032041 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.031935 4413 generic.go:356] "Generic (PLEG): container finished" podID="edbd4ad4-8205-460e-9385-3151a16e7b68" containerID="84b38ee5b78874b3ba63aea701531b75db4e36b58104379752c623bb528cf2b8" exitCode=0 Oct 30 11:54:07.032041 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.031943 4413 generic.go:356] "Generic (PLEG): container finished" podID="edbd4ad4-8205-460e-9385-3151a16e7b68" containerID="363ecfdfc38f12a1827471bf130dfc3cf81020d3f74ae830c5a715e79fdad493" exitCode=0 Oct 30 11:54:07.032041 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.031950 4413 generic.go:356] "Generic (PLEG): container finished" podID="edbd4ad4-8205-460e-9385-3151a16e7b68" containerID="ad81f53518fc97beb0249bddf2f7dc7694fc11d429e8197076968900fabeb112" exitCode=0 Oct 30 11:54:07.032041 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.031955 4413 generic.go:356] "Generic (PLEG): container finished" podID="edbd4ad4-8205-460e-9385-3151a16e7b68" containerID="30609759de1232481a500ece957c0d7ced169b67c5e10d98044ed22480d668a0" exitCode=0 Oct 30 11:54:07.039265 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.039232 4413 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:54:07.039447 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.039428 4413 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 11:54:07.060581 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.060555 4413 manager.go:324] Recovery completed Oct 30 11:54:07.074602 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.074581 4413 cpu_manager.go:226] "Starting" policy="none" Oct 30 11:54:07.074602 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.074595 4413 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 11:54:07.074740 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.074613 4413 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 11:54:07.074740 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.074724 4413 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 11:54:07.074808 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.074735 4413 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 11:54:07.074808 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.074751 4413 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 11:54:07.074808 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.074756 4413 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 11:54:07.074808 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.074761 4413 policy_none.go:50] "Start" Oct 30 11:54:07.074808 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.074776 4413 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 11:54:07.074808 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.074785 4413 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 11:54:07.074962 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.074903 4413 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 11:54:07.074962 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.074910 4413 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 11:54:07.074962 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.074915 4413 policy_none.go:44] "Start" Oct 30 11:54:07.078927 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.078897 4413 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 11:54:07.083896 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.083876 4413 manager.go:341] "Starting Device Plugin manager" Oct 30 11:54:07.084744 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:07.083913 4413 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 11:54:07.084826 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.084749 4413 server.go:87] "Starting device plugin registration server" Oct 30 11:54:07.085021 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.085001 4413 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 11:54:07.085121 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.085018 4413 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 11:54:07.085121 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.085108 4413 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 11:54:07.085262 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.085208 4413 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 11:54:07.085324 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.085273 4413 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 11:54:07.085324 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.085282 4413 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 11:54:07.085505 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.085354 4413 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T11:54:07.08526232Z","UUID":"97b83b76-dd50-4be8-921c-f174ffde3e0b","Handler":null,"Name":"","Endpoint":""} Oct 30 11:54:07.086717 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:07.086557 4413 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 11:54:07.086819 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.086803 4413 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 11:54:07.087154 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.087139 4413 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 11:54:07.112426 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.112380 4413 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal"] Oct 30 11:54:07.112850 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.112823 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" event={"ID":"3e9967fa0ac01d9b4f8c399a65ef8efb","Type":"ContainerStarted","Data":"bad257fab8bc8ef59d6258631ff37ef3fb6e9004612587df680aa1ec5dcbe9a5"} Oct 30 11:54:07.112893 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.112859 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" event={"ID":"3e9967fa0ac01d9b4f8c399a65ef8efb","Type":"ContainerDied","Data":"f8e5a280c6002d9ccb9d53c330f406f1fe5f8714903135b2ef2f4e517d5f9b4e"} Oct 30 11:54:07.112893 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.112883 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" event={"ID":"3e9967fa0ac01d9b4f8c399a65ef8efb","Type":"ContainerStarted","Data":"17b883f17b8183ee1f8d45b16f43ee7d69bd5f0cf23fac2fbf8fff150588df71"} Oct 30 11:54:07.112951 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.112905 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" event={"ID":"8314998578de81aba897c7d170293e92","Type":"ContainerStarted","Data":"864076f4ae7bae8cf58ceb926c5533e31c730d2280ec8f65cbeb9655711967f0"} Oct 30 11:54:07.112951 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.112925 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" event={"ID":"8314998578de81aba897c7d170293e92","Type":"ContainerStarted","Data":"376ea76ae9c666fdffe54b57c639f33b6068a1b0f6df2c719aa300065016161e"} Oct 30 11:54:07.112951 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.112937 4413 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:54:07.113086 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.113072 4413 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" Oct 30 11:54:07.136049 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.136029 4413 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:54:07.136140 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:07.136066 4413 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-23.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" Oct 30 11:54:07.136586 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.136573 4413 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:54:07.136621 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:07.136607 4413 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:54:07.186091 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.186066 4413 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 11:54:07.187330 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.187312 4413 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientMemory" Oct 30 11:54:07.187407 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.187342 4413 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 11:54:07.187407 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.187353 4413 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-23.ec2.internal" event="NodeHasSufficientPID" Oct 30 11:54:07.187407 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.187383 4413 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-23.ec2.internal" Oct 30 11:54:07.223659 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.223635 4413 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-23.ec2.internal" Oct 30 11:54:07.223767 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.223699 4413 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-23.ec2.internal" Oct 30 11:54:07.227417 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.227398 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8314998578de81aba897c7d170293e92-config\") pod \"kube-apiserver-proxy-ip-10-0-138-23.ec2.internal\" (UID: \"8314998578de81aba897c7d170293e92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" Oct 30 11:54:07.227496 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.227423 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e9967fa0ac01d9b4f8c399a65ef8efb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" (UID: \"3e9967fa0ac01d9b4f8c399a65ef8efb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:54:07.227496 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.227442 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e9967fa0ac01d9b4f8c399a65ef8efb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" (UID: \"3e9967fa0ac01d9b4f8c399a65ef8efb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:54:07.328280 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.328185 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e9967fa0ac01d9b4f8c399a65ef8efb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" (UID: \"3e9967fa0ac01d9b4f8c399a65ef8efb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:54:07.328280 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.328215 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e9967fa0ac01d9b4f8c399a65ef8efb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" (UID: \"3e9967fa0ac01d9b4f8c399a65ef8efb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:54:07.328458 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.328281 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e9967fa0ac01d9b4f8c399a65ef8efb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" (UID: \"3e9967fa0ac01d9b4f8c399a65ef8efb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:54:07.328458 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.328301 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8314998578de81aba897c7d170293e92-config\") pod \"kube-apiserver-proxy-ip-10-0-138-23.ec2.internal\" (UID: \"8314998578de81aba897c7d170293e92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" Oct 30 11:54:07.328458 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.328337 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8314998578de81aba897c7d170293e92-config\") pod \"kube-apiserver-proxy-ip-10-0-138-23.ec2.internal\" (UID: \"8314998578de81aba897c7d170293e92\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" Oct 30 11:54:07.328458 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.328371 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e9967fa0ac01d9b4f8c399a65ef8efb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" (UID: \"3e9967fa0ac01d9b4f8c399a65ef8efb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:54:07.965491 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:07.965456 4413 apiserver.go:52] "Watching apiserver" Oct 30 11:54:08.004432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.004406 4413 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 11:54:08.007082 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.005361 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-gshq6","kube-system/global-pull-secret-syncer-pnvz9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6","openshift-multus/network-metrics-daemon-qvs98","openshift-image-registry/node-ca-f64ck","openshift-network-diagnostics/network-check-target-g584t","kube-system/konnectivity-agent-7lb7d","openshift-dns/node-resolver-jz62m","openshift-dns/dns-default-lqnn8","openshift-ingress-canary/ingress-canary-lfqp6","openshift-insights/insights-runtime-extractor-bdxnx","openshift-kube-storage-version-migrator/migrator-66df5d496b-7r9dz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-kwcqk","kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal","openshift-network-operator/iptables-alerter-s96m4","openshift-cluster-node-tuning-operator/tuned-dmj7l","openshift-multus/multus-additional-cni-plugins-hchnk"] Oct 30 11:54:08.007738 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.007717 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:54:08.012225 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.012206 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lqnn8" Oct 30 11:54:08.013030 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.013013 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:08.023431 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.023409 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 11:54:08.023508 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.023495 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 11:54:08.023574 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.023560 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 11:54:08.023632 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.023587 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 11:54:08.023632 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.023608 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 11:54:08.024388 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.024364 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 11:54:08.024714 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.024700 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6hs8x\"" Oct 30 11:54:08.024797 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.024785 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-llkx2\"" Oct 30 11:54:08.024986 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.024970 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jjcjh\"" Oct 30 11:54:08.025172 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.025159 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 11:54:08.025298 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.025278 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zfblq\"" Oct 30 11:54:08.025466 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.025450 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 11:54:08.025515 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.025475 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pxw6w\"" Oct 30 11:54:08.025675 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.025648 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 11:54:08.025675 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.025662 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rpjw7\"" Oct 30 11:54:08.025675 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.025653 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 11:54:08.025675 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.025653 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 11:54:08.025857 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.025726 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7cm7r\"" Oct 30 11:54:08.035034 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.035013 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 11:54:08.035111 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.035091 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 11:54:08.035111 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.035102 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 11:54:08.035784 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.035758 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 11:54:08.035841 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.035798 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 11:54:08.035900 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.035866 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lrxbh\"" Oct 30 11:54:08.035900 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.035889 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 11:54:08.035988 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.035914 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 11:54:08.036059 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036046 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 11:54:08.036114 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036064 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjl8c\"" Oct 30 11:54:08.036114 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036100 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tz47t\"" Oct 30 11:54:08.036463 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036436 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 11:54:08.036463 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036456 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 11:54:08.036598 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036479 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 11:54:08.036598 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036499 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 11:54:08.036598 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036516 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:54:08.036598 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036525 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 11:54:08.036598 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036528 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 11:54:08.036598 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036551 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 11:54:08.036598 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036578 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 11:54:08.036598 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036583 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8n4dr\"" Oct 30 11:54:08.036598 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036501 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 11:54:08.037111 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036605 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 11:54:08.037111 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036636 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 11:54:08.037111 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036638 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 11:54:08.037111 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036674 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 11:54:08.037111 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036683 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tdxpc\"" Oct 30 11:54:08.037111 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036693 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sjprl\"" Oct 30 11:54:08.037111 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036696 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:54:08.037111 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036806 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxkrq\"" Oct 30 11:54:08.037111 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036821 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 11:54:08.037111 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.036992 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8pg5x\"" Oct 30 11:54:08.037111 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.037034 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sw2hc\"" Oct 30 11:54:08.037111 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.037066 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 11:54:08.037585 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.037154 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 11:54:08.037585 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.037180 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 11:54:08.037585 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.037267 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 11:54:08.037585 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.037292 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 11:54:08.037585 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.037378 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 11:54:08.037585 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.037474 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 11:54:08.050189 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.050174 4413 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" Oct 30 11:54:08.050376 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.050309 4413 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:54:08.071678 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.071661 4413 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:54:08.071730 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:08.071696 4413 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-23.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-23.ec2.internal" Oct 30 11:54:08.072233 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.072220 4413 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 11:54:08.072289 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:08.072269 4413 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-23.ec2.internal" Oct 30 11:54:08.079918 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.079892 4413 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 11:54:08.134579 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.134557 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edbd4ad4-8205-460e-9385-3151a16e7b68-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:54:08.134701 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.134589 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-log-socket\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.134701 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.134606 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg275\" (UniqueName: \"kubernetes.io/projected/82e6695b-1b88-4372-9d00-ddf341253c59-kube-api-access-kg275\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:08.134701 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.134625 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-socket-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.134850 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.134696 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc833d39-b781-486b-9fcc-549cba555ed9-tmp\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.134850 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.134733 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-system-cni-dir\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.134850 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.134763 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edbd4ad4-8205-460e-9385-3151a16e7b68-cnibin\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:54:08.134850 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.134783 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc833d39-b781-486b-9fcc-549cba555ed9-tmp\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.134850 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.134786 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edbd4ad4-8205-460e-9385-3151a16e7b68-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:54:08.135024 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.134863 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-run-openvswitch\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.135024 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.134883 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-cni-netd\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.135024 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.134897 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.135024 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.134912 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-run\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.135024 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.134927 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-hostroot\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.135024 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.134975 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/864715b0-07ec-483a-8401-089c747422cb-multus-daemon-config\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.135024 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135009 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-etc-kubernetes\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.135319 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135081 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edbd4ad4-8205-460e-9385-3151a16e7b68-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:54:08.135319 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135114 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edbd4ad4-8205-460e-9385-3151a16e7b68-system-cni-dir\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:54:08.135319 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135145 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-device-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.135319 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135172 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw9h2\" (UniqueName: \"kubernetes.io/projected/27f694f1-e1de-483f-a5c5-250f52dae768-kube-api-access-bw9h2\") pod \"node-ca-f64ck\" (UID: \"27f694f1-e1de-483f-a5c5-250f52dae768\") " pod="openshift-image-registry/node-ca-f64ck" Oct 30 11:54:08.135319 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135229 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/864715b0-07ec-483a-8401-089c747422cb-multus-daemon-config\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.135319 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135241 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a3afa91-aff6-455b-aeda-7275cd8ac8e0-agent-certs\") pod \"konnectivity-agent-7lb7d\" (UID: \"1a3afa91-aff6-455b-aeda-7275cd8ac8e0\") " pod="kube-system/konnectivity-agent-7lb7d" Oct 30 11:54:08.135319 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135309 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edbd4ad4-8205-460e-9385-3151a16e7b68-cni-binary-copy\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:54:08.135583 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135400 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-slash\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.135583 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135496 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-var-lib-openvswitch\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.135583 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135523 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a3afa91-aff6-455b-aeda-7275cd8ac8e0-agent-certs\") pod \"konnectivity-agent-7lb7d\" (UID: \"1a3afa91-aff6-455b-aeda-7275cd8ac8e0\") " pod="kube-system/konnectivity-agent-7lb7d" Oct 30 11:54:08.135583 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135527 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-run-ovn\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.135583 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135544 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edbd4ad4-8205-460e-9385-3151a16e7b68-cni-binary-copy\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:54:08.135583 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135552 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-sysctl-d\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.135858 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135594 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-cnibin\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.135858 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135626 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-var-lib-cni-bin\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.135858 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135661 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbv7k\" (UniqueName: \"kubernetes.io/projected/52ede1dc-817e-434c-874a-2db2a9c10f52-kube-api-access-kbv7k\") pod \"iptables-alerter-s96m4\" (UID: \"52ede1dc-817e-434c-874a-2db2a9c10f52\") " pod="openshift-network-operator/iptables-alerter-s96m4" Oct 30 11:54:08.135858 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135693 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-node-log\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.135858 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135731 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82e6695b-1b88-4372-9d00-ddf341253c59-crio-socket\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:08.135858 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135749 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs\") pod \"network-metrics-daemon-qvs98\" (UID: \"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd\") " pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:54:08.135858 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135795 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-run-netns\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.135858 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135843 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5wc8\" (UniqueName: \"kubernetes.io/projected/125e1073-73c6-4aae-9c1b-4dc6e047e241-kube-api-access-h5wc8\") pod \"ingress-canary-lfqp6\" (UID: \"125e1073-73c6-4aae-9c1b-4dc6e047e241\") " pod="openshift-ingress-canary/ingress-canary-lfqp6" Oct 30 11:54:08.136230 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.135919 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-sys-fs\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.136230 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136011 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz6mk\" (UniqueName: \"kubernetes.io/projected/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-kube-api-access-tz6mk\") pod \"network-metrics-daemon-qvs98\" (UID: \"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd\") " pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:54:08.136230 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136035 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc833d39-b781-486b-9fcc-549cba555ed9-etc-tuned\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.136230 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136053 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-kubelet-config\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:54:08.136230 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136070 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-cni-bin\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.136230 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136099 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:54:08.136230 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136118 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-registration-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.136230 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136146 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-systemd\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.136230 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136149 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fc833d39-b781-486b-9fcc-549cba555ed9-etc-tuned\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.136230 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136169 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-run-netns\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.136230 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136215 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-var-lib-kubelet\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.136726 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136238 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/535eda1f-d9e7-4908-b958-56f531b68fbb-hosts-file\") pod \"node-resolver-jz62m\" (UID: \"535eda1f-d9e7-4908-b958-56f531b68fbb\") " pod="openshift-dns/node-resolver-jz62m" Oct 30 11:54:08.136726 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136279 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c3ac2bd-4472-4779-8ca5-460160cdedbb-ovnkube-config\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.136726 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136310 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-sys\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.136726 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136342 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-multus-socket-dir-parent\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.136726 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136452 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-kubelet\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.136726 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136484 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c3ac2bd-4472-4779-8ca5-460160cdedbb-ovnkube-config\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.136726 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136486 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c3ac2bd-4472-4779-8ca5-460160cdedbb-ovn-node-metrics-cert\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.136726 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136530 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27f694f1-e1de-483f-a5c5-250f52dae768-serviceca\") pod \"node-ca-f64ck\" (UID: \"27f694f1-e1de-483f-a5c5-250f52dae768\") " pod="openshift-image-registry/node-ca-f64ck" Oct 30 11:54:08.136726 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136556 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-modprobe-d\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.136726 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136581 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-var-lib-cni-multus\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.136726 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136608 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c3ac2bd-4472-4779-8ca5-460160cdedbb-ovn-node-metrics-cert\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.136726 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136729 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-multus-conf-dir\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136749 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nqkz\" (UniqueName: \"kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz\") pod \"network-check-target-g584t\" (UID: \"8b4fd8cb-a92d-40b2-8133-b04c126332e7\") " pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136764 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/125e1073-73c6-4aae-9c1b-4dc6e047e241-cert\") pod \"ingress-canary-lfqp6\" (UID: \"125e1073-73c6-4aae-9c1b-4dc6e047e241\") " pod="openshift-ingress-canary/ingress-canary-lfqp6" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136780 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-etc-selinux\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136795 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-run-systemd\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136829 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82e6695b-1b88-4372-9d00-ddf341253c59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136808 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27f694f1-e1de-483f-a5c5-250f52dae768-serviceca\") pod \"node-ca-f64ck\" (UID: \"27f694f1-e1de-483f-a5c5-250f52dae768\") " pod="openshift-image-registry/node-ca-f64ck" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136874 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136919 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2vgd\" (UniqueName: \"kubernetes.io/projected/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-kube-api-access-m2vgd\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136943 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-kubernetes\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136968 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-var-lib-kubelet\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.136995 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/125e1073-73c6-4aae-9c1b-4dc6e047e241-cert\") pod \"ingress-canary-lfqp6\" (UID: \"125e1073-73c6-4aae-9c1b-4dc6e047e241\") " pod="openshift-ingress-canary/ingress-canary-lfqp6" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137016 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a3afa91-aff6-455b-aeda-7275cd8ac8e0-konnectivity-ca\") pod \"konnectivity-agent-7lb7d\" (UID: \"1a3afa91-aff6-455b-aeda-7275cd8ac8e0\") " pod="kube-system/konnectivity-agent-7lb7d" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137047 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vkxp\" (UniqueName: \"kubernetes.io/projected/535eda1f-d9e7-4908-b958-56f531b68fbb-kube-api-access-2vkxp\") pod \"node-resolver-jz62m\" (UID: \"535eda1f-d9e7-4908-b958-56f531b68fbb\") " pod="openshift-dns/node-resolver-jz62m" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137087 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhm5c\" (UniqueName: \"kubernetes.io/projected/edbd4ad4-8205-460e-9385-3151a16e7b68-kube-api-access-hhm5c\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137118 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c3ac2bd-4472-4779-8ca5-460160cdedbb-ovnkube-script-lib\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137143 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27f694f1-e1de-483f-a5c5-250f52dae768-host\") pod \"node-ca-f64ck\" (UID: \"27f694f1-e1de-483f-a5c5-250f52dae768\") " pod="openshift-image-registry/node-ca-f64ck" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137169 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edbd4ad4-8205-460e-9385-3151a16e7b68-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137172 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a3afa91-aff6-455b-aeda-7275cd8ac8e0-konnectivity-ca\") pod \"konnectivity-agent-7lb7d\" (UID: \"1a3afa91-aff6-455b-aeda-7275cd8ac8e0\") " pod="kube-system/konnectivity-agent-7lb7d" Oct 30 11:54:08.137185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137194 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137226 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-config-volume\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137268 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-sysconfig\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137292 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-sysctl-conf\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137317 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-lib-modules\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137343 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-systemd-units\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137377 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c3ac2bd-4472-4779-8ca5-460160cdedbb-ovnkube-script-lib\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137382 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-etc-openvswitch\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137396 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edbd4ad4-8205-460e-9385-3151a16e7b68-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137412 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137447 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c3ac2bd-4472-4779-8ca5-460160cdedbb-env-overrides\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137462 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-config-volume\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137477 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82e6695b-1b88-4372-9d00-ddf341253c59-data-volume\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137555 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c3ac2bd-4472-4779-8ca5-460160cdedbb-env-overrides\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137576 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-tmp-dir\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137615 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82e6695b-1b88-4372-9d00-ddf341253c59-data-volume\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137646 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-run-k8s-cni-cncf-io\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137663 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-tmp-dir\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137678 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/52ede1dc-817e-434c-874a-2db2a9c10f52-iptables-alerter-script\") pod \"iptables-alerter-s96m4\" (UID: \"52ede1dc-817e-434c-874a-2db2a9c10f52\") " pod="openshift-network-operator/iptables-alerter-s96m4" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137694 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/535eda1f-d9e7-4908-b958-56f531b68fbb-tmp-dir\") pod \"node-resolver-jz62m\" (UID: \"535eda1f-d9e7-4908-b958-56f531b68fbb\") " pod="openshift-dns/node-resolver-jz62m" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137712 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-host\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137731 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/535eda1f-d9e7-4908-b958-56f531b68fbb-tmp-dir\") pod \"node-resolver-jz62m\" (UID: \"535eda1f-d9e7-4908-b958-56f531b68fbb\") " pod="openshift-dns/node-resolver-jz62m" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137730 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzcmw\" (UniqueName: \"kubernetes.io/projected/fc833d39-b781-486b-9fcc-549cba555ed9-kube-api-access-hzcmw\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137754 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:54:08.137842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137834 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-multus-cni-dir\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.138735 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137918 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-run-multus-certs\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.138735 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137959 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7dwl\" (UniqueName: \"kubernetes.io/projected/864715b0-07ec-483a-8401-089c747422cb-kube-api-access-w7dwl\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.138735 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137962 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/52ede1dc-817e-434c-874a-2db2a9c10f52-iptables-alerter-script\") pod \"iptables-alerter-s96m4\" (UID: \"52ede1dc-817e-434c-874a-2db2a9c10f52\") " pod="openshift-network-operator/iptables-alerter-s96m4" Oct 30 11:54:08.138735 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.137987 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-dbus\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:54:08.138735 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.138013 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-original-pull-secret\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:54:08.138735 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.138082 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brmjx\" (UniqueName: \"kubernetes.io/projected/1c3ac2bd-4472-4779-8ca5-460160cdedbb-kube-api-access-brmjx\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.138735 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.138105 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhwsn\" (UniqueName: \"kubernetes.io/projected/39ef6d56-2baa-4f93-9268-12a74e0ef903-kube-api-access-fhwsn\") pod \"migrator-66df5d496b-7r9dz\" (UID: \"39ef6d56-2baa-4f93-9268-12a74e0ef903\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7r9dz" Oct 30 11:54:08.138735 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.138276 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx4gc\" (UniqueName: \"kubernetes.io/projected/e2b6ec57-b883-408a-ab01-abd192572510-kube-api-access-vx4gc\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.138735 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.138307 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/864715b0-07ec-483a-8401-089c747422cb-cni-binary-copy\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.138735 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.138332 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52ede1dc-817e-434c-874a-2db2a9c10f52-host-slash\") pod \"iptables-alerter-s96m4\" (UID: \"52ede1dc-817e-434c-874a-2db2a9c10f52\") " pod="openshift-network-operator/iptables-alerter-s96m4" Oct 30 11:54:08.138735 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.138494 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/864715b0-07ec-483a-8401-089c747422cb-cni-binary-copy\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.155500 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.155474 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg275\" (UniqueName: \"kubernetes.io/projected/82e6695b-1b88-4372-9d00-ddf341253c59-kube-api-access-kg275\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:08.155924 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.155908 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw9h2\" (UniqueName: \"kubernetes.io/projected/27f694f1-e1de-483f-a5c5-250f52dae768-kube-api-access-bw9h2\") pod \"node-ca-f64ck\" (UID: \"27f694f1-e1de-483f-a5c5-250f52dae768\") " pod="openshift-image-registry/node-ca-f64ck" Oct 30 11:54:08.156343 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.156325 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbv7k\" (UniqueName: \"kubernetes.io/projected/52ede1dc-817e-434c-874a-2db2a9c10f52-kube-api-access-kbv7k\") pod \"iptables-alerter-s96m4\" (UID: \"52ede1dc-817e-434c-874a-2db2a9c10f52\") " pod="openshift-network-operator/iptables-alerter-s96m4" Oct 30 11:54:08.156430 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.156403 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz6mk\" (UniqueName: \"kubernetes.io/projected/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-kube-api-access-tz6mk\") pod \"network-metrics-daemon-qvs98\" (UID: \"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd\") " pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:54:08.156530 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.156515 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5wc8\" (UniqueName: \"kubernetes.io/projected/125e1073-73c6-4aae-9c1b-4dc6e047e241-kube-api-access-h5wc8\") pod \"ingress-canary-lfqp6\" (UID: \"125e1073-73c6-4aae-9c1b-4dc6e047e241\") " pod="openshift-ingress-canary/ingress-canary-lfqp6" Oct 30 11:54:08.163240 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.163210 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nqkz\" (UniqueName: \"kubernetes.io/projected/8b4fd8cb-a92d-40b2-8133-b04c126332e7-kube-api-access-9nqkz\") pod \"network-check-target-g584t\" (UID: \"8b4fd8cb-a92d-40b2-8133-b04c126332e7\") " pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:54:08.163376 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.163312 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2vgd\" (UniqueName: \"kubernetes.io/projected/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-kube-api-access-m2vgd\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:54:08.163376 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.163325 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhwsn\" (UniqueName: \"kubernetes.io/projected/39ef6d56-2baa-4f93-9268-12a74e0ef903-kube-api-access-fhwsn\") pod \"migrator-66df5d496b-7r9dz\" (UID: \"39ef6d56-2baa-4f93-9268-12a74e0ef903\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7r9dz" Oct 30 11:54:08.163376 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.163370 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7dwl\" (UniqueName: \"kubernetes.io/projected/864715b0-07ec-483a-8401-089c747422cb-kube-api-access-w7dwl\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.163551 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.163430 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vkxp\" (UniqueName: \"kubernetes.io/projected/535eda1f-d9e7-4908-b958-56f531b68fbb-kube-api-access-2vkxp\") pod \"node-resolver-jz62m\" (UID: \"535eda1f-d9e7-4908-b958-56f531b68fbb\") " pod="openshift-dns/node-resolver-jz62m" Oct 30 11:54:08.163551 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.163504 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx4gc\" (UniqueName: \"kubernetes.io/projected/e2b6ec57-b883-408a-ab01-abd192572510-kube-api-access-vx4gc\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.163551 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.163516 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhm5c\" (UniqueName: \"kubernetes.io/projected/edbd4ad4-8205-460e-9385-3151a16e7b68-kube-api-access-hhm5c\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:54:08.163863 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.163839 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzcmw\" (UniqueName: \"kubernetes.io/projected/fc833d39-b781-486b-9fcc-549cba555ed9-kube-api-access-hzcmw\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.164004 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.163990 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brmjx\" (UniqueName: \"kubernetes.io/projected/1c3ac2bd-4472-4779-8ca5-460160cdedbb-kube-api-access-brmjx\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.239400 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239327 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27f694f1-e1de-483f-a5c5-250f52dae768-host\") pod \"node-ca-f64ck\" (UID: \"27f694f1-e1de-483f-a5c5-250f52dae768\") " pod="openshift-image-registry/node-ca-f64ck" Oct 30 11:54:08.239400 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239365 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.239400 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239391 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-sysconfig\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.239622 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239414 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-sysctl-conf\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.239622 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239436 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-lib-modules\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.239622 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239438 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27f694f1-e1de-483f-a5c5-250f52dae768-host\") pod \"node-ca-f64ck\" (UID: \"27f694f1-e1de-483f-a5c5-250f52dae768\") " pod="openshift-image-registry/node-ca-f64ck" Oct 30 11:54:08.239622 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239455 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-systemd-units\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.239622 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239455 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.239622 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239479 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-etc-openvswitch\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.239622 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239531 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.239622 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239558 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-run-k8s-cni-cncf-io\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.239622 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239579 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-lib-modules\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.239622 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239592 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-sysctl-conf\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.239622 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239583 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-host\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.239622 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239623 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-host\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239632 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-multus-cni-dir\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239636 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239475 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-sysconfig\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239657 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-run-multus-certs\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239680 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-dbus\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239691 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-systemd-units\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239659 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-run-k8s-cni-cncf-io\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239706 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-multus-cni-dir\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239707 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52ede1dc-817e-434c-874a-2db2a9c10f52-host-slash\") pod \"iptables-alerter-s96m4\" (UID: \"52ede1dc-817e-434c-874a-2db2a9c10f52\") " pod="openshift-network-operator/iptables-alerter-s96m4" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239719 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-etc-openvswitch\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239730 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-run-multus-certs\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239743 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52ede1dc-817e-434c-874a-2db2a9c10f52-host-slash\") pod \"iptables-alerter-s96m4\" (UID: \"52ede1dc-817e-434c-874a-2db2a9c10f52\") " pod="openshift-network-operator/iptables-alerter-s96m4" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239747 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-log-socket\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239738 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-dbus\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239781 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-log-socket\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239802 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-socket-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239829 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-system-cni-dir\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239849 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edbd4ad4-8205-460e-9385-3151a16e7b68-cnibin\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239865 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edbd4ad4-8205-460e-9385-3151a16e7b68-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239867 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-system-cni-dir\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239879 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-run-openvswitch\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239894 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-cni-netd\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239895 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edbd4ad4-8205-460e-9385-3151a16e7b68-cnibin\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239832 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-socket-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239909 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239922 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edbd4ad4-8205-460e-9385-3151a16e7b68-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239936 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-run-openvswitch\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239934 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-run\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239945 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-cni-netd\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239954 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239966 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-hostroot\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239989 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-etc-kubernetes\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.239994 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-run\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240014 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edbd4ad4-8205-460e-9385-3151a16e7b68-system-cni-dir\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240026 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-hostroot\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240037 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-device-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240102 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-device-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.240099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240034 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-etc-kubernetes\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240144 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-slash\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240168 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edbd4ad4-8205-460e-9385-3151a16e7b68-system-cni-dir\") pod \"multus-additional-cni-plugins-hchnk\" (UID: \"edbd4ad4-8205-460e-9385-3151a16e7b68\") " pod="openshift-multus/multus-additional-cni-plugins-hchnk" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240208 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-var-lib-openvswitch\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240231 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-slash\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240278 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-run-ovn\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240304 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-run-ovn\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240333 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-sysctl-d\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240347 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-cnibin\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240463 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-sysctl-d\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240467 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-var-lib-openvswitch\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240466 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-cnibin\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240488 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-var-lib-cni-bin\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240517 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-node-log\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240534 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82e6695b-1b88-4372-9d00-ddf341253c59-crio-socket\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240560 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs\") pod \"network-metrics-daemon-qvs98\" (UID: \"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd\") " pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240583 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-node-log\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240585 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-run-netns\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240612 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82e6695b-1b88-4372-9d00-ddf341253c59-crio-socket\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240618 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-run-netns\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240563 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-var-lib-cni-bin\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240623 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-sys-fs\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240666 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-sys-fs\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240678 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-kubelet-config\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240719 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-cni-bin\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240730 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/68aaec5d-0bc6-4e8b-a311-1797a055bb7b-kubelet-config\") pod \"global-pull-secret-syncer-pnvz9\" (UID: \"68aaec5d-0bc6-4e8b-a311-1797a055bb7b\") " pod="kube-system/global-pull-secret-syncer-pnvz9" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240741 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240757 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-cni-bin\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240763 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-registration-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240784 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-systemd\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240841 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-systemd\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:08.240836 4413 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240865 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-registration-dir\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240898 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-run-netns\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:08.240917 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls podName:62adf1a5-f87e-4cbc-86e5-bcd2cefbd390 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:08.740902249 +0000 UTC m=+1.850745140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls") pod "dns-default-lqnn8" (UID: "62adf1a5-f87e-4cbc-86e5-bcd2cefbd390") : secret "dns-default-metrics-tls" not found Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240933 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-run-netns\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240943 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-var-lib-kubelet\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240974 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-var-lib-kubelet\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240983 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/535eda1f-d9e7-4908-b958-56f531b68fbb-hosts-file\") pod \"node-resolver-jz62m\" (UID: \"535eda1f-d9e7-4908-b958-56f531b68fbb\") " pod="openshift-dns/node-resolver-jz62m" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241000 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-sys\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241016 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-multus-socket-dir-parent\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241018 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/535eda1f-d9e7-4908-b958-56f531b68fbb-hosts-file\") pod \"node-resolver-jz62m\" (UID: \"535eda1f-d9e7-4908-b958-56f531b68fbb\") " pod="openshift-dns/node-resolver-jz62m" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.240997 4413 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241048 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-kubelet\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241054 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-multus-socket-dir-parent\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241071 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-modprobe-d\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241084 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-sys\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241094 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-var-lib-cni-multus\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241108 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-host-kubelet\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241118 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-multus-conf-dir\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241142 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-etc-selinux\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241152 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-host-var-lib-cni-multus\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241158 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/864715b0-07ec-483a-8401-089c747422cb-multus-conf-dir\") pod \"multus-gshq6\" (UID: \"864715b0-07ec-483a-8401-089c747422cb\") " pod="openshift-multus/multus-gshq6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241165 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-modprobe-d\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241157 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-run-systemd\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241198 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e2b6ec57-b883-408a-ab01-abd192572510-etc-selinux\") pod \"aws-ebs-csi-driver-node-lcrs6\" (UID: \"e2b6ec57-b883-408a-ab01-abd192572510\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lcrs6" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241207 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82e6695b-1b88-4372-9d00-ddf341253c59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241199 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c3ac2bd-4472-4779-8ca5-460160cdedbb-run-systemd\") pod \"ovnkube-node-kwcqk\" (UID: \"1c3ac2bd-4472-4779-8ca5-460160cdedbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241226 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241266 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-kubernetes\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241282 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-var-lib-kubelet\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:08.241324 4413 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241356 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-etc-kubernetes\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:08.241372 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls podName:82e6695b-1b88-4372-9d00-ddf341253c59 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:08.741355757 +0000 UTC m=+1.851198634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdxnx" (UID: "82e6695b-1b88-4372-9d00-ddf341253c59") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:08.241718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241391 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc833d39-b781-486b-9fcc-549cba555ed9-var-lib-kubelet\") pod \"tuned-dmj7l\" (UID: \"fc833d39-b781-486b-9fcc-549cba555ed9\") " pod="openshift-cluster-node-tuning-operator/tuned-dmj7l" Oct 30 11:54:08.243451 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.241786 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82e6695b-1b88-4372-9d00-ddf341253c59-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:08.243451 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.243211 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd-metrics-certs\") pod \"network-metrics-daemon-qvs98\" (UID: \"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd\") " pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:54:08.308557 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.308530 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvs98" Oct 30 11:54:08.442657 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.442624 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:08.466325 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.466301 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-7r9dz_39ef6d56-2baa-4f93-9268-12a74e0ef903/migrator/0.log" Oct 30 11:54:08.653867 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.653832 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvs98"] Oct 30 11:54:08.657204 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:08.657178 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd5adf9_b6c1_4faa_9ac7_f3329d5478cd.slice/crio-3843c905ef8c256d3913181264089d9f4cd71dc40693e838d4002cb38999cffe WatchSource:0}: Error finding container 3843c905ef8c256d3913181264089d9f4cd71dc40693e838d4002cb38999cffe: Status 404 returned error can't find the container with id 3843c905ef8c256d3913181264089d9f4cd71dc40693e838d4002cb38999cffe Oct 30 11:54:08.659427 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.659412 4413 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 11:54:08.744330 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.744293 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:54:08.744502 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.744342 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:08.744502 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:08.744437 4413 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:08.744502 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:08.744442 4413 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:54:08.744502 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:08.744497 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls podName:82e6695b-1b88-4372-9d00-ddf341253c59 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:09.744484286 +0000 UTC m=+2.854327149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdxnx" (UID: "82e6695b-1b88-4372-9d00-ddf341253c59") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:08.744665 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:08.744510 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls podName:62adf1a5-f87e-4cbc-86e5-bcd2cefbd390 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:09.744504265 +0000 UTC m=+2.854347123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls") pod "dns-default-lqnn8" (UID: "62adf1a5-f87e-4cbc-86e5-bcd2cefbd390") : secret "dns-default-metrics-tls" not found Oct 30 11:54:08.936348 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:08.936268 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:09.053595 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:09.053550 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvs98" event={"ID":"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd","Type":"ContainerStarted","Data":"3843c905ef8c256d3913181264089d9f4cd71dc40693e838d4002cb38999cffe"} Oct 30 11:54:09.060369 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:09.060325 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:09.191592 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:09.191508 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:09.317587 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:09.317506 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:09.433499 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:09.433469 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:09.647834 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:09.647803 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:09.751193 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:09.751147 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:54:09.751361 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:09.751203 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:09.751361 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:09.751312 4413 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 11:54:09.751459 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:09.751381 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls podName:62adf1a5-f87e-4cbc-86e5-bcd2cefbd390 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:11.751365496 +0000 UTC m=+4.861208356 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls") pod "dns-default-lqnn8" (UID: "62adf1a5-f87e-4cbc-86e5-bcd2cefbd390") : secret "dns-default-metrics-tls" not found Oct 30 11:54:09.751459 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:09.751315 4413 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:09.751459 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:09.751457 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls podName:82e6695b-1b88-4372-9d00-ddf341253c59 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:11.751441164 +0000 UTC m=+4.861284031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdxnx" (UID: "82e6695b-1b88-4372-9d00-ddf341253c59") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:09.878043 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:09.878008 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:10.438493 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:10.438461 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:11.038056 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:11.038022 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:11.059916 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:11.059887 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvs98" event={"ID":"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd","Type":"ContainerStarted","Data":"c4841dd35381fab994164552b8ded37e27fc727e4ca404c2c9e8b1279096ffa1"} Oct 30 11:54:11.060008 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:11.059941 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvs98" event={"ID":"4cd5adf9-b6c1-4faa-9ac7-f3329d5478cd","Type":"ContainerStarted","Data":"440245afb5ae2bc5a65091bd06e67423dcde366e4ef337c24ae3579fbf86484c"} Oct 30 11:54:11.239583 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:11.239534 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:11.440646 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:11.440616 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:11.761641 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:11.761545 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:11.761641 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:11.761604 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:54:11.761832 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:11.761708 4413 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 11:54:11.761832 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:11.761777 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls podName:82e6695b-1b88-4372-9d00-ddf341253c59 nodeName:}" failed. No retries permitted until 2026-10-30 11:54:15.761760668 +0000 UTC m=+8.871603530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bdxnx" (UID: "82e6695b-1b88-4372-9d00-ddf341253c59") : secret "insights-runtime-extractor-tls" not found Oct 30 11:54:11.764031 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:11.764012 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/62adf1a5-f87e-4cbc-86e5-bcd2cefbd390-metrics-tls\") pod \"dns-default-lqnn8\" (UID: \"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390\") " pod="openshift-dns/dns-default-lqnn8" Oct 30 11:54:11.791081 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:11.791051 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:11.913482 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:11.913442 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lqnn8" Oct 30 11:54:11.920024 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:11.919969 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:12.206725 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:12.206700 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lqnn8"] Oct 30 11:54:12.209721 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:12.209696 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62adf1a5_f87e_4cbc_86e5_bcd2cefbd390.slice/crio-e8048f53920fd59043e79ada36f4cdc7ad4e3deb77fe28a589b8e5072c7e5d7b WatchSource:0}: Error finding container e8048f53920fd59043e79ada36f4cdc7ad4e3deb77fe28a589b8e5072c7e5d7b: Status 404 returned error can't find the container with id e8048f53920fd59043e79ada36f4cdc7ad4e3deb77fe28a589b8e5072c7e5d7b Oct 30 11:54:12.633755 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:12.633692 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:12.750691 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:12.750661 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:12.893133 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:12.893030 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:13.039158 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:13.039123 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:13.065145 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:13.065110 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lqnn8" event={"ID":"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390","Type":"ContainerStarted","Data":"e8048f53920fd59043e79ada36f4cdc7ad4e3deb77fe28a589b8e5072c7e5d7b"} Oct 30 11:54:13.240859 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:13.240755 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:13.369239 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:13.369202 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:13.493312 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:13.493227 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:13.766694 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:13.766666 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:13.767040 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:13.766875 4413 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 11:54:13.855126 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:13.855094 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:14.072058 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:14.071974 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lqnn8" event={"ID":"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390","Type":"ContainerStarted","Data":"4a61ffe449aac9315b4858fb2b02ad9a27b466b73baccc09f63b0fbb38ce4621"} Oct 30 11:54:14.072058 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:14.072030 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lqnn8" event={"ID":"62adf1a5-f87e-4cbc-86e5-bcd2cefbd390","Type":"ContainerStarted","Data":"4114faf419464a9e9f602790e00934b27ddf79a17dccbb9c6fc1c862e5d90a96"} Oct 30 11:54:14.072228 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:14.072115 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lqnn8" Oct 30 11:54:14.646733 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:14.646701 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:15.047980 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:15.047903 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:15.236877 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:15.236850 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:15.453794 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:15.453763 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:15.638040 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:15.638016 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:54:15.642883 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:15.642854 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:15.789834 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:15.789755 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:15.792174 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:15.792149 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82e6695b-1b88-4372-9d00-ddf341253c59-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bdxnx\" (UID: \"82e6695b-1b88-4372-9d00-ddf341253c59\") " pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:15.814090 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:15.814060 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bdxnx" Oct 30 11:54:16.128639 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:16.128610 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bdxnx"] Oct 30 11:54:16.131770 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:16.131745 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e6695b_1b88_4372_9d00_ddf341253c59.slice/crio-2a4d6923829e2267fc0f31d1dc353b6e4dc8f1b13223abebf949a7f722d78a75 WatchSource:0}: Error finding container 2a4d6923829e2267fc0f31d1dc353b6e4dc8f1b13223abebf949a7f722d78a75: Status 404 returned error can't find the container with id 2a4d6923829e2267fc0f31d1dc353b6e4dc8f1b13223abebf949a7f722d78a75 Oct 30 11:54:17.082535 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:17.082501 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bdxnx" event={"ID":"82e6695b-1b88-4372-9d00-ddf341253c59","Type":"ContainerStarted","Data":"c3a4f0b82417a25d578af92b2da4aca7c18430f2c6540fad89ca8271ce2aaebf"} Oct 30 11:54:17.082690 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:17.082555 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bdxnx" event={"ID":"82e6695b-1b88-4372-9d00-ddf341253c59","Type":"ContainerStarted","Data":"2b4fcdd3d37293e62ba23f9de62cc8451261aa462c70bbee856df5cda18a33d0"} Oct 30 11:54:17.082690 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:17.082573 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bdxnx" event={"ID":"82e6695b-1b88-4372-9d00-ddf341253c59","Type":"ContainerStarted","Data":"2a4d6923829e2267fc0f31d1dc353b6e4dc8f1b13223abebf949a7f722d78a75"} Oct 30 11:54:19.091186 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:19.091147 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bdxnx" event={"ID":"82e6695b-1b88-4372-9d00-ddf341253c59","Type":"ContainerStarted","Data":"d8512002d59c386ca3fabcb0eb0f763adbd75a226a92a5b9d36c46953812a8aa"} Oct 30 11:54:23.047646 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:23.047616 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:23.280603 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:23.280571 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:23.397099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:23.397075 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:24.077866 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:24.077838 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lqnn8" Oct 30 11:54:25.470969 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:25.470920 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:25.618559 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:25.618491 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:25.758327 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:25.758204 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:28.245418 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:28.245385 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:29.929918 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:29.929883 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:30.114713 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.114678 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:30.242329 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.242233 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:30.325005 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.324973 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9"] Oct 30 11:54:30.327824 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.327799 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.364051 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.364026 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 11:54:30.364051 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.364029 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 11:54:30.364223 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.364081 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 11:54:30.364223 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.364086 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 11:54:30.364727 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.364709 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9"] Oct 30 11:54:30.376006 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.375985 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bgk7n"] Oct 30 11:54:30.378363 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.378350 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.415273 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.415239 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 11:54:30.415803 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.415790 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 11:54:30.415885 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.415871 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 11:54:30.416044 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.416033 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 11:54:30.416093 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.416041 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zmtdp\"" Oct 30 11:54:30.416093 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.416041 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xwvts\"" Oct 30 11:54:30.417947 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.417926 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:30.452063 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.452041 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 11:54:30.486476 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.486452 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/18e30a28-a964-429e-8305-f5ffbe40ba3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.486569 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.486486 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/18e30a28-a964-429e-8305-f5ffbe40ba3b-node-exporter-tls\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.486569 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.486503 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/18e30a28-a964-429e-8305-f5ffbe40ba3b-node-exporter-wtmp\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.486569 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.486557 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18e30a28-a964-429e-8305-f5ffbe40ba3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.486683 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.486576 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea0bc518-3942-4980-8d19-dfd59954169a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9q6m9\" (UID: \"ea0bc518-3942-4980-8d19-dfd59954169a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.486683 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.486626 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea0bc518-3942-4980-8d19-dfd59954169a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9q6m9\" (UID: \"ea0bc518-3942-4980-8d19-dfd59954169a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.486683 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.486675 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/18e30a28-a964-429e-8305-f5ffbe40ba3b-root\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.486790 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.486724 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0bc518-3942-4980-8d19-dfd59954169a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9q6m9\" (UID: \"ea0bc518-3942-4980-8d19-dfd59954169a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.486790 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.486767 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfh2l\" (UniqueName: \"kubernetes.io/projected/ea0bc518-3942-4980-8d19-dfd59954169a-kube-api-access-qfh2l\") pod \"kube-state-metrics-5dc8d7d89b-9q6m9\" (UID: \"ea0bc518-3942-4980-8d19-dfd59954169a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.486865 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.486854 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea0bc518-3942-4980-8d19-dfd59954169a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9q6m9\" (UID: \"ea0bc518-3942-4980-8d19-dfd59954169a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.486894 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.486877 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/18e30a28-a964-429e-8305-f5ffbe40ba3b-node-exporter-textfile\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.486937 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.486929 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18e30a28-a964-429e-8305-f5ffbe40ba3b-sys\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.486971 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.486955 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl8bq\" (UniqueName: \"kubernetes.io/projected/18e30a28-a964-429e-8305-f5ffbe40ba3b-kube-api-access-nl8bq\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.487009 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.486998 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18e30a28-a964-429e-8305-f5ffbe40ba3b-metrics-client-ca\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.487046 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.487014 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea0bc518-3942-4980-8d19-dfd59954169a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9q6m9\" (UID: \"ea0bc518-3942-4980-8d19-dfd59954169a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.587843 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.587772 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0bc518-3942-4980-8d19-dfd59954169a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9q6m9\" (UID: \"ea0bc518-3942-4980-8d19-dfd59954169a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.587843 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.587804 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfh2l\" (UniqueName: \"kubernetes.io/projected/ea0bc518-3942-4980-8d19-dfd59954169a-kube-api-access-qfh2l\") pod \"kube-state-metrics-5dc8d7d89b-9q6m9\" (UID: \"ea0bc518-3942-4980-8d19-dfd59954169a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.587843 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.587830 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea0bc518-3942-4980-8d19-dfd59954169a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9q6m9\" (UID: \"ea0bc518-3942-4980-8d19-dfd59954169a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.588072 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.587848 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/18e30a28-a964-429e-8305-f5ffbe40ba3b-node-exporter-textfile\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.588072 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.587865 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18e30a28-a964-429e-8305-f5ffbe40ba3b-sys\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.588072 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.587883 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl8bq\" (UniqueName: \"kubernetes.io/projected/18e30a28-a964-429e-8305-f5ffbe40ba3b-kube-api-access-nl8bq\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.588072 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.587912 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18e30a28-a964-429e-8305-f5ffbe40ba3b-metrics-client-ca\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.588072 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.587936 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea0bc518-3942-4980-8d19-dfd59954169a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9q6m9\" (UID: \"ea0bc518-3942-4980-8d19-dfd59954169a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.588072 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.587945 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18e30a28-a964-429e-8305-f5ffbe40ba3b-sys\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.588072 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.587983 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/18e30a28-a964-429e-8305-f5ffbe40ba3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.588072 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.588017 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/18e30a28-a964-429e-8305-f5ffbe40ba3b-node-exporter-tls\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.588072 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.588034 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/18e30a28-a964-429e-8305-f5ffbe40ba3b-node-exporter-wtmp\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.588072 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.588054 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18e30a28-a964-429e-8305-f5ffbe40ba3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.588072 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.588076 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea0bc518-3942-4980-8d19-dfd59954169a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9q6m9\" (UID: \"ea0bc518-3942-4980-8d19-dfd59954169a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.588634 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.588105 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea0bc518-3942-4980-8d19-dfd59954169a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9q6m9\" (UID: \"ea0bc518-3942-4980-8d19-dfd59954169a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.588634 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.588129 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/18e30a28-a964-429e-8305-f5ffbe40ba3b-root\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.588634 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.588193 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/18e30a28-a964-429e-8305-f5ffbe40ba3b-root\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.588634 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.588374 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/18e30a28-a964-429e-8305-f5ffbe40ba3b-node-exporter-textfile\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.588634 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.588440 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/18e30a28-a964-429e-8305-f5ffbe40ba3b-node-exporter-wtmp\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.588634 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.588446 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea0bc518-3942-4980-8d19-dfd59954169a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9q6m9\" (UID: \"ea0bc518-3942-4980-8d19-dfd59954169a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.588950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.588743 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0bc518-3942-4980-8d19-dfd59954169a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9q6m9\" (UID: \"ea0bc518-3942-4980-8d19-dfd59954169a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.588950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.588760 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18e30a28-a964-429e-8305-f5ffbe40ba3b-metrics-client-ca\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.588950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.588916 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea0bc518-3942-4980-8d19-dfd59954169a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9q6m9\" (UID: \"ea0bc518-3942-4980-8d19-dfd59954169a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.589336 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.589314 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/18e30a28-a964-429e-8305-f5ffbe40ba3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.590779 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.590758 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/18e30a28-a964-429e-8305-f5ffbe40ba3b-node-exporter-tls\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.590905 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.590825 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18e30a28-a964-429e-8305-f5ffbe40ba3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.590905 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.590868 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea0bc518-3942-4980-8d19-dfd59954169a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9q6m9\" (UID: \"ea0bc518-3942-4980-8d19-dfd59954169a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.590975 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.590960 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea0bc518-3942-4980-8d19-dfd59954169a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9q6m9\" (UID: \"ea0bc518-3942-4980-8d19-dfd59954169a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.634160 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.634131 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfh2l\" (UniqueName: \"kubernetes.io/projected/ea0bc518-3942-4980-8d19-dfd59954169a-kube-api-access-qfh2l\") pod \"kube-state-metrics-5dc8d7d89b-9q6m9\" (UID: \"ea0bc518-3942-4980-8d19-dfd59954169a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.634817 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.634799 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl8bq\" (UniqueName: \"kubernetes.io/projected/18e30a28-a964-429e-8305-f5ffbe40ba3b-kube-api-access-nl8bq\") pod \"node-exporter-bgk7n\" (UID: \"18e30a28-a964-429e-8305-f5ffbe40ba3b\") " pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.640722 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.640699 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" Oct 30 11:54:30.647682 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.647656 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:30.690681 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.690655 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bgk7n" Oct 30 11:54:30.707705 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:30.707666 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e30a28_a964_429e_8305_f5ffbe40ba3b.slice/crio-fe6647f426d7ee881c4914d7fbc58310e986fe2935dc9da7c15666da64cac53a WatchSource:0}: Error finding container fe6647f426d7ee881c4914d7fbc58310e986fe2935dc9da7c15666da64cac53a: Status 404 returned error can't find the container with id fe6647f426d7ee881c4914d7fbc58310e986fe2935dc9da7c15666da64cac53a Oct 30 11:54:30.826003 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.825970 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:30.918845 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.918814 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7lb7d" Oct 30 11:54:30.945882 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:30.945857 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9"] Oct 30 11:54:30.949293 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:30.949241 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0bc518_3942_4980_8d19_dfd59954169a.slice/crio-5832cfc89f5bccc2a5edcecdcab9713ab250ac87eba70321068e3946a2762585 WatchSource:0}: Error finding container 5832cfc89f5bccc2a5edcecdcab9713ab250ac87eba70321068e3946a2762585: Status 404 returned error can't find the container with id 5832cfc89f5bccc2a5edcecdcab9713ab250ac87eba70321068e3946a2762585 Oct 30 11:54:31.002228 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.002193 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:31.126369 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.126333 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bgk7n" event={"ID":"18e30a28-a964-429e-8305-f5ffbe40ba3b","Type":"ContainerStarted","Data":"fe6647f426d7ee881c4914d7fbc58310e986fe2935dc9da7c15666da64cac53a"} Oct 30 11:54:31.127483 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.127462 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" event={"ID":"ea0bc518-3942-4980-8d19-dfd59954169a","Type":"ContainerStarted","Data":"5832cfc89f5bccc2a5edcecdcab9713ab250ac87eba70321068e3946a2762585"} Oct 30 11:54:31.313874 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.313790 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:31.441505 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.441473 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:31.579124 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.579053 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 11:54:31.582014 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.581987 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:31.582456 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.582430 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.603273 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.601787 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 11:54:31.603273 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.602766 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 11:54:31.603273 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.602823 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 11:54:31.603273 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.602873 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 11:54:31.603273 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.602943 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m2ldd\"" Oct 30 11:54:31.603273 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.602986 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 11:54:31.603273 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.603065 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 11:54:31.603273 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.603127 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 11:54:31.603273 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.603182 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 11:54:31.627154 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.627127 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 11:54:31.628460 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.628436 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 11:54:31.697869 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.697841 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0fc60ce-3b21-4f0c-84f0-58f627675180-config-out\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.697991 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.697886 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0fc60ce-3b21-4f0c-84f0-58f627675180-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.697991 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.697918 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0fc60ce-3b21-4f0c-84f0-58f627675180-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.698112 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.698031 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0fc60ce-3b21-4f0c-84f0-58f627675180-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.698112 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.698096 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.698191 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.698125 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.698225 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.698187 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn84j\" (UniqueName: \"kubernetes.io/projected/d0fc60ce-3b21-4f0c-84f0-58f627675180-kube-api-access-qn84j\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.698225 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.698222 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-web-config\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.698356 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.698244 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.698356 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.698347 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-config-volume\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.698458 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.698406 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.698458 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.698435 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0fc60ce-3b21-4f0c-84f0-58f627675180-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.698545 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.698464 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.799621 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.799592 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-config-volume\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.799749 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.799648 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.799749 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.799672 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0fc60ce-3b21-4f0c-84f0-58f627675180-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.799749 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.799699 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.799749 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.799734 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0fc60ce-3b21-4f0c-84f0-58f627675180-config-out\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.799945 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.799758 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0fc60ce-3b21-4f0c-84f0-58f627675180-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.799945 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.799787 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0fc60ce-3b21-4f0c-84f0-58f627675180-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.799945 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.799814 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0fc60ce-3b21-4f0c-84f0-58f627675180-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.799945 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.799848 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.799945 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.799877 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.799945 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.799901 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn84j\" (UniqueName: \"kubernetes.io/projected/d0fc60ce-3b21-4f0c-84f0-58f627675180-kube-api-access-qn84j\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.799945 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.799936 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-web-config\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.800274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.799971 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.802455 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.801199 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0fc60ce-3b21-4f0c-84f0-58f627675180-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.802455 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.801880 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0fc60ce-3b21-4f0c-84f0-58f627675180-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.803361 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.803013 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0fc60ce-3b21-4f0c-84f0-58f627675180-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.803361 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.803125 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.804218 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.804193 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-web-config\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.804686 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.804661 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.805134 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.805113 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-config-volume\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.805557 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.805537 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.805871 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.805832 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0fc60ce-3b21-4f0c-84f0-58f627675180-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.805947 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.805873 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.806374 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.806355 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0fc60ce-3b21-4f0c-84f0-58f627675180-config-out\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.807038 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.807020 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.826908 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.826875 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn84j\" (UniqueName: \"kubernetes.io/projected/d0fc60ce-3b21-4f0c-84f0-58f627675180-kube-api-access-qn84j\") pod \"alertmanager-main-0\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:31.920496 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:31.920458 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:54:32.132429 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.132400 4413 generic.go:356] "Generic (PLEG): container finished" podID="18e30a28-a964-429e-8305-f5ffbe40ba3b" containerID="91375d3da51f59481fe69d45ac0a1ea8b448db1664e157276423822fe01818af" exitCode=0 Oct 30 11:54:32.132826 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.132439 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bgk7n" event={"ID":"18e30a28-a964-429e-8305-f5ffbe40ba3b","Type":"ContainerDied","Data":"91375d3da51f59481fe69d45ac0a1ea8b448db1664e157276423822fe01818af"} Oct 30 11:54:32.272842 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.272821 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 11:54:32.276618 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:32.276594 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0fc60ce_3b21_4f0c_84f0_58f627675180.slice/crio-1e6a5270aa2b6b9c2a7eae78e643a021e2e7136a799a28ace303980ee4fc963b WatchSource:0}: Error finding container 1e6a5270aa2b6b9c2a7eae78e643a021e2e7136a799a28ace303980ee4fc963b: Status 404 returned error can't find the container with id 1e6a5270aa2b6b9c2a7eae78e643a021e2e7136a799a28ace303980ee4fc963b Oct 30 11:54:32.437016 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.435818 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8676c985b4-fm9dg"] Oct 30 11:54:32.452101 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.452063 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.465289 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.463318 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8676c985b4-fm9dg"] Oct 30 11:54:32.469616 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.469582 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 11:54:32.469616 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.469614 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-frhgp8cr5bt1o\"" Oct 30 11:54:32.469838 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.469680 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 11:54:32.469838 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.469798 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 11:54:32.469838 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.469814 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4r96f\"" Oct 30 11:54:32.469838 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.469572 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 11:54:32.470020 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.469877 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 11:54:32.603897 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.603865 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.604041 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.603901 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-metrics-client-ca\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.604041 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.603922 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-tmp\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.604112 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.604059 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.604112 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.604101 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-secret-thanos-querier-tls\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.604173 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.604164 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-225n2\" (UniqueName: \"kubernetes.io/projected/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-kube-api-access-225n2\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.604207 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.604197 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.604239 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.604217 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.604313 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.604300 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-secret-grpc-tls\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.704976 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.704887 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-metrics-client-ca\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.704976 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.704933 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-tmp\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.704976 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.704963 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.705293 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.704997 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-secret-thanos-querier-tls\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.705293 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.705125 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-225n2\" (UniqueName: \"kubernetes.io/projected/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-kube-api-access-225n2\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.705293 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.705189 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.705451 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.705351 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.705507 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.705460 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-tmp\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.705507 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.705465 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-secret-grpc-tls\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.705598 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.705577 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.706022 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.705952 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-metrics-client-ca\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.708666 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.708630 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.708787 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.708684 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.709316 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.709296 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-secret-thanos-querier-tls\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.709388 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.709367 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-secret-grpc-tls\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.719437 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.719405 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.719561 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.719405 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.729297 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.729236 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-225n2\" (UniqueName: \"kubernetes.io/projected/c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1-kube-api-access-225n2\") pod \"thanos-querier-8676c985b4-fm9dg\" (UID: \"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1\") " pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:32.850629 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:32.850600 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:33.052322 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:33.052226 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:33.138122 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:33.138083 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" event={"ID":"ea0bc518-3942-4980-8d19-dfd59954169a","Type":"ContainerStarted","Data":"ac91fe4e5245075da3de0f8d64caf70a629c95c8a8e88dd95fad0633f75d812a"} Oct 30 11:54:33.138610 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:33.138180 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" event={"ID":"ea0bc518-3942-4980-8d19-dfd59954169a","Type":"ContainerStarted","Data":"f3241e8dc3961df332d5d2cd17b3348cdabcaa42af6544c7e84001a3f600ae82"} Oct 30 11:54:33.138610 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:33.138211 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" event={"ID":"ea0bc518-3942-4980-8d19-dfd59954169a","Type":"ContainerStarted","Data":"2bf5d1e5a9cb9eb2b55d440e383a465bd7366a803a9cb755e8791f6c3fb53fdd"} Oct 30 11:54:33.139600 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:33.139570 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0fc60ce-3b21-4f0c-84f0-58f627675180","Type":"ContainerStarted","Data":"1e6a5270aa2b6b9c2a7eae78e643a021e2e7136a799a28ace303980ee4fc963b"} Oct 30 11:54:33.142179 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:33.142154 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bgk7n" event={"ID":"18e30a28-a964-429e-8305-f5ffbe40ba3b","Type":"ContainerStarted","Data":"4a1e22274980146c7b38acadd6edec66699f30980adcdf49c186a2d69df35dfa"} Oct 30 11:54:33.142306 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:33.142198 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bgk7n" event={"ID":"18e30a28-a964-429e-8305-f5ffbe40ba3b","Type":"ContainerStarted","Data":"02769e199377c8eb6acbad465bfe4d5b178777fe7940aaab4fa8cc197004dde3"} Oct 30 11:54:33.166716 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:33.166689 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:33.176908 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:33.176886 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8676c985b4-fm9dg"] Oct 30 11:54:33.180920 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:33.180893 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d0a2b0_8520_448b_aa9f_cd4e43f84ff1.slice/crio-1197db18f3ffc783097b1f590480fb872ccab43c57dd5d059271c8f00f7c6610 WatchSource:0}: Error finding container 1197db18f3ffc783097b1f590480fb872ccab43c57dd5d059271c8f00f7c6610: Status 404 returned error can't find the container with id 1197db18f3ffc783097b1f590480fb872ccab43c57dd5d059271c8f00f7c6610 Oct 30 11:54:33.188732 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:33.188682 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9q6m9" podStartSLOduration=1.9138215939999998 podStartE2EDuration="3.18866775s" totalImagesPullingTime="1.274846156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:33.183434767 +0000 UTC m=+26.293277648" watchObservedRunningTime="2026-10-30 11:54:33.18866775 +0000 UTC m=+26.298510633" Oct 30 11:54:33.231221 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:33.231162 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bgk7n" podStartSLOduration=2.228516262 podStartE2EDuration="3.231144995s" totalImagesPullingTime="918.078777ms" totalInitContainerRuntime="84.549956ms" isStatefulPod=true podCreationTimestamp="2026-10-30 11:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:33.229409004 +0000 UTC m=+26.339251886" watchObservedRunningTime="2026-10-30 11:54:33.231144995 +0000 UTC m=+26.340987877" Oct 30 11:54:33.301197 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:33.301166 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:33.431083 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:33.431052 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:33.546874 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:33.546842 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:33.661951 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:33.661925 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:34.078226 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:34.078196 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwcqk" Oct 30 11:54:34.147787 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:34.147749 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" event={"ID":"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1","Type":"ContainerStarted","Data":"1197db18f3ffc783097b1f590480fb872ccab43c57dd5d059271c8f00f7c6610"} Oct 30 11:54:34.149857 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:34.149721 4413 generic.go:356] "Generic (PLEG): container finished" podID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerID="a37438c3128d51a22760328efd1dc02f86cfdc26a9203c7a2d0b4312438d7862" exitCode=0 Oct 30 11:54:34.149857 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:34.149751 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0fc60ce-3b21-4f0c-84f0-58f627675180","Type":"ContainerDied","Data":"a37438c3128d51a22760328efd1dc02f86cfdc26a9203c7a2d0b4312438d7862"} Oct 30 11:54:34.949536 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:34.949500 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-94cc96cf4-mq2jm"] Oct 30 11:54:34.951995 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:34.951970 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:34.967943 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:34.967918 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 11:54:34.968115 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:34.968083 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 11:54:34.979724 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:34.979700 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 11:54:34.980015 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:34.979977 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 11:54:34.980166 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:34.980150 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nq469\"" Oct 30 11:54:34.980270 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:34.980240 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-25fitrot8iele\"" Oct 30 11:54:34.981129 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:34.981112 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-94cc96cf4-mq2jm"] Oct 30 11:54:35.030910 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.030880 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rqhdc"] Oct 30 11:54:35.033385 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.033363 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqhdc" Oct 30 11:54:35.042566 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.042529 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:35.054119 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.054055 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rqhdc"] Oct 30 11:54:35.066478 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.066448 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 11:54:35.066599 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.066573 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-v75hs\"" Oct 30 11:54:35.128674 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.128638 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f122aa99-4d33-450f-acd4-da132dd1cb2e-audit-log\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.128855 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.128709 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f122aa99-4d33-450f-acd4-da132dd1cb2e-client-ca-bundle\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.128855 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.128739 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f122aa99-4d33-450f-acd4-da132dd1cb2e-secret-metrics-server-client-certs\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.128855 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.128763 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g8dg\" (UniqueName: \"kubernetes.io/projected/f122aa99-4d33-450f-acd4-da132dd1cb2e-kube-api-access-8g8dg\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.128855 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.128795 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f122aa99-4d33-450f-acd4-da132dd1cb2e-metrics-server-audit-profiles\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.128855 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.128851 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f122aa99-4d33-450f-acd4-da132dd1cb2e-secret-metrics-server-tls\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.129218 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.128874 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f122aa99-4d33-450f-acd4-da132dd1cb2e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.158589 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.158553 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:35.229796 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.229770 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tm5j\" (UniqueName: \"kubernetes.io/projected/59d12821-d129-4883-89c6-219a359e3143-kube-api-access-7tm5j\") pod \"monitoring-plugin-77f4f76594-rqhdc\" (UID: \"59d12821-d129-4883-89c6-219a359e3143\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqhdc" Oct 30 11:54:35.229937 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.229826 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f122aa99-4d33-450f-acd4-da132dd1cb2e-secret-metrics-server-tls\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.229999 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.229960 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f122aa99-4d33-450f-acd4-da132dd1cb2e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.230054 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.230007 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f122aa99-4d33-450f-acd4-da132dd1cb2e-audit-log\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.230054 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.230040 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59d12821-d129-4883-89c6-219a359e3143-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rqhdc\" (UID: \"59d12821-d129-4883-89c6-219a359e3143\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqhdc" Oct 30 11:54:35.230160 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.230080 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f122aa99-4d33-450f-acd4-da132dd1cb2e-client-ca-bundle\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.230160 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.230109 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f122aa99-4d33-450f-acd4-da132dd1cb2e-secret-metrics-server-client-certs\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.230160 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.230133 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g8dg\" (UniqueName: \"kubernetes.io/projected/f122aa99-4d33-450f-acd4-da132dd1cb2e-kube-api-access-8g8dg\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.230337 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.230200 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f122aa99-4d33-450f-acd4-da132dd1cb2e-metrics-server-audit-profiles\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.230630 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.230604 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f122aa99-4d33-450f-acd4-da132dd1cb2e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.230958 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.230940 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f122aa99-4d33-450f-acd4-da132dd1cb2e-audit-log\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.231024 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.230987 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f122aa99-4d33-450f-acd4-da132dd1cb2e-metrics-server-audit-profiles\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.233344 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.233327 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f122aa99-4d33-450f-acd4-da132dd1cb2e-client-ca-bundle\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.233595 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.233549 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f122aa99-4d33-450f-acd4-da132dd1cb2e-secret-metrics-server-client-certs\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.234329 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.234305 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f122aa99-4d33-450f-acd4-da132dd1cb2e-secret-metrics-server-tls\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.254804 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.254774 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g8dg\" (UniqueName: \"kubernetes.io/projected/f122aa99-4d33-450f-acd4-da132dd1cb2e-kube-api-access-8g8dg\") pod \"metrics-server-94cc96cf4-mq2jm\" (UID: \"f122aa99-4d33-450f-acd4-da132dd1cb2e\") " pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.268220 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.268197 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:35.332237 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.331542 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59d12821-d129-4883-89c6-219a359e3143-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rqhdc\" (UID: \"59d12821-d129-4883-89c6-219a359e3143\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqhdc" Oct 30 11:54:35.332237 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.331621 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tm5j\" (UniqueName: \"kubernetes.io/projected/59d12821-d129-4883-89c6-219a359e3143-kube-api-access-7tm5j\") pod \"monitoring-plugin-77f4f76594-rqhdc\" (UID: \"59d12821-d129-4883-89c6-219a359e3143\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqhdc" Oct 30 11:54:35.336719 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.336505 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/59d12821-d129-4883-89c6-219a359e3143-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rqhdc\" (UID: \"59d12821-d129-4883-89c6-219a359e3143\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqhdc" Oct 30 11:54:35.359479 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.359370 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tm5j\" (UniqueName: \"kubernetes.io/projected/59d12821-d129-4883-89c6-219a359e3143-kube-api-access-7tm5j\") pod \"monitoring-plugin-77f4f76594-rqhdc\" (UID: \"59d12821-d129-4883-89c6-219a359e3143\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqhdc" Oct 30 11:54:35.565727 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.565648 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-65459c448-trpdm"] Oct 30 11:54:35.568716 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.568695 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.596824 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:35.596798 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf122aa99_4d33_450f_acd4_da132dd1cb2e.slice/crio-6c8017828a0b6a1a5ee50a323d8afb1faf737ea16b1dbcc872a71307a08ed53a WatchSource:0}: Error finding container 6c8017828a0b6a1a5ee50a323d8afb1faf737ea16b1dbcc872a71307a08ed53a: Status 404 returned error can't find the container with id 6c8017828a0b6a1a5ee50a323d8afb1faf737ea16b1dbcc872a71307a08ed53a Oct 30 11:54:35.598870 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.598844 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 11:54:35.598870 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.598852 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 11:54:35.599503 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.598882 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 11:54:35.599503 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.598958 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 11:54:35.599503 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.598979 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xtzmn\"" Oct 30 11:54:35.599503 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.599125 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 11:54:35.609274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.609228 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 11:54:35.610458 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.610427 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-94cc96cf4-mq2jm"] Oct 30 11:54:35.611230 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.611215 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65459c448-trpdm"] Oct 30 11:54:35.640682 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.640653 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g584t" Oct 30 11:54:35.647181 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.647148 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqhdc" Oct 30 11:54:35.739367 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.736805 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2f8e0b61-2169-4d89-8315-4092541ac795-secret-telemeter-client\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.739367 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.736861 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f8e0b61-2169-4d89-8315-4092541ac795-metrics-client-ca\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.739367 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.736892 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f8e0b61-2169-4d89-8315-4092541ac795-serving-certs-ca-bundle\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.739367 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.737272 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2f8e0b61-2169-4d89-8315-4092541ac795-telemeter-client-tls\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.739367 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.737338 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fhcd\" (UniqueName: \"kubernetes.io/projected/2f8e0b61-2169-4d89-8315-4092541ac795-kube-api-access-9fhcd\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.739367 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.737379 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f8e0b61-2169-4d89-8315-4092541ac795-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.739367 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.737440 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2f8e0b61-2169-4d89-8315-4092541ac795-federate-client-tls\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.739367 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.737472 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f8e0b61-2169-4d89-8315-4092541ac795-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.819225 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:54:35.819143 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 11:54:35.838370 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.838338 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2f8e0b61-2169-4d89-8315-4092541ac795-federate-client-tls\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.838546 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.838392 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f8e0b61-2169-4d89-8315-4092541ac795-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.838546 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.838429 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2f8e0b61-2169-4d89-8315-4092541ac795-secret-telemeter-client\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.838546 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.838460 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f8e0b61-2169-4d89-8315-4092541ac795-metrics-client-ca\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.838718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.838585 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f8e0b61-2169-4d89-8315-4092541ac795-serving-certs-ca-bundle\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.838718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.838630 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2f8e0b61-2169-4d89-8315-4092541ac795-telemeter-client-tls\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.838718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.838666 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fhcd\" (UniqueName: \"kubernetes.io/projected/2f8e0b61-2169-4d89-8315-4092541ac795-kube-api-access-9fhcd\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.838718 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.838697 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f8e0b61-2169-4d89-8315-4092541ac795-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.839479 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.839385 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f8e0b61-2169-4d89-8315-4092541ac795-metrics-client-ca\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.839933 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.839913 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f8e0b61-2169-4d89-8315-4092541ac795-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.840040 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.840006 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f8e0b61-2169-4d89-8315-4092541ac795-serving-certs-ca-bundle\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.841213 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.841186 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f8e0b61-2169-4d89-8315-4092541ac795-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.841336 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.841291 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2f8e0b61-2169-4d89-8315-4092541ac795-telemeter-client-tls\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.841383 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.841372 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2f8e0b61-2169-4d89-8315-4092541ac795-federate-client-tls\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.841420 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.841397 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2f8e0b61-2169-4d89-8315-4092541ac795-secret-telemeter-client\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.862437 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.862412 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fhcd\" (UniqueName: \"kubernetes.io/projected/2f8e0b61-2169-4d89-8315-4092541ac795-kube-api-access-9fhcd\") pod \"telemeter-client-65459c448-trpdm\" (UID: \"2f8e0b61-2169-4d89-8315-4092541ac795\") " pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.884831 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.884817 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65459c448-trpdm" Oct 30 11:54:35.955380 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:35.955287 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rqhdc"] Oct 30 11:54:36.166291 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:36.166017 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0fc60ce-3b21-4f0c-84f0-58f627675180","Type":"ContainerStarted","Data":"516002f8c38f4c394a1dae1fade992bb05187f42544eb44b5b536f46988a1cb1"} Oct 30 11:54:36.166291 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:36.166098 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0fc60ce-3b21-4f0c-84f0-58f627675180","Type":"ContainerStarted","Data":"e65896f9acc7585c182199cbb1dc2baee5489d22fa00b5b783b819626a7f2a51"} Oct 30 11:54:36.166291 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:36.166134 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0fc60ce-3b21-4f0c-84f0-58f627675180","Type":"ContainerStarted","Data":"f83251e509d39fcf471e1612945ba49ccd466243c0695496158457e7e1f0058c"} Oct 30 11:54:36.166291 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:36.166170 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0fc60ce-3b21-4f0c-84f0-58f627675180","Type":"ContainerStarted","Data":"ce3be1474d3a55b658527f293c4fb8baec9abeaff0234d0bb7326f881f7c0e2b"} Oct 30 11:54:36.166291 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:36.166215 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0fc60ce-3b21-4f0c-84f0-58f627675180","Type":"ContainerStarted","Data":"d9e715992c746509bcb7a6172e3dd55b1b3623ce16ba0cc3d55095bba24488d7"} Oct 30 11:54:36.170148 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:36.170116 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" event={"ID":"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1","Type":"ContainerStarted","Data":"677b78e01bfa59f696a10d18b4c29356db8efa4ef9dc13228aeea4bffec89a41"} Oct 30 11:54:36.170800 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:36.170785 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" event={"ID":"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1","Type":"ContainerStarted","Data":"134a3a3724d71bad6b915b7f2c466d03b9074266bd712bc82c1af3c368b7b447"} Oct 30 11:54:36.171081 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:36.171069 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" event={"ID":"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1","Type":"ContainerStarted","Data":"df2c6123f7b93b46948c274c94a866e9fb3b8304622cd6b8141d29c8f514c459"} Oct 30 11:54:36.171397 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:36.171352 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqhdc" event={"ID":"59d12821-d129-4883-89c6-219a359e3143","Type":"ContainerStarted","Data":"226371bcca0bdc1fbf36fe445fe35f79c2d9b05a06db42139b1f15e7d3091749"} Oct 30 11:54:36.172796 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:36.172747 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" event={"ID":"f122aa99-4d33-450f-acd4-da132dd1cb2e","Type":"ContainerStarted","Data":"6c8017828a0b6a1a5ee50a323d8afb1faf737ea16b1dbcc872a71307a08ed53a"} Oct 30 11:54:36.199421 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:36.199393 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65459c448-trpdm"] Oct 30 11:54:36.203177 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:36.203153 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8e0b61_2169_4d89_8315_4092541ac795.slice/crio-26781d67da248d0558fbe1d3a6fc5fe76003333719772e9b6bdb05d55dbae1bc WatchSource:0}: Error finding container 26781d67da248d0558fbe1d3a6fc5fe76003333719772e9b6bdb05d55dbae1bc: Status 404 returned error can't find the container with id 26781d67da248d0558fbe1d3a6fc5fe76003333719772e9b6bdb05d55dbae1bc Oct 30 11:54:37.171667 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.171639 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 11:54:37.175134 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.175112 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.186176 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.186151 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:37.192085 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.190054 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 11:54:37.192085 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.190167 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 11:54:37.192085 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.190235 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 11:54:37.192085 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.190322 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 11:54:37.192085 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.190495 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0fc60ce-3b21-4f0c-84f0-58f627675180","Type":"ContainerStarted","Data":"3099befc61e1d442366a385f55a50d1f9ee2940162d918fcc8d1a84b93b7ac45"} Oct 30 11:54:37.197189 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.197151 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 11:54:37.197796 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.197467 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 11:54:37.197796 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.197525 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 11:54:37.197796 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.197524 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 11:54:37.197796 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.197639 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 11:54:37.197796 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.197716 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-l2pdp\"" Oct 30 11:54:37.197796 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.197759 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7dn3dgofmkf17\"" Oct 30 11:54:37.197796 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.197763 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 11:54:37.198172 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.198018 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65459c448-trpdm" event={"ID":"2f8e0b61-2169-4d89-8315-4092541ac795","Type":"ContainerStarted","Data":"26781d67da248d0558fbe1d3a6fc5fe76003333719772e9b6bdb05d55dbae1bc"} Oct 30 11:54:37.201066 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.201044 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 11:54:37.203580 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.203547 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" event={"ID":"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1","Type":"ContainerStarted","Data":"54879015b191afca7befc1387b1963d752dcc6b13d53c3748204ad2eb30af160"} Oct 30 11:54:37.203674 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.203617 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" event={"ID":"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1","Type":"ContainerStarted","Data":"f54ed651425c469347538ade759942bf6f99fba03836da0a8f5ffc8cc2dde61d"} Oct 30 11:54:37.203674 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.203658 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" event={"ID":"c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1","Type":"ContainerStarted","Data":"39a92ca7d3c9a322e0fbbb93c729269d0a087b09dfdc05b3d290849fd028b54e"} Oct 30 11:54:37.204411 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.204391 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:37.223176 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.223149 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 11:54:37.223791 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.223762 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 11:54:37.229056 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.229024 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 11:54:37.230699 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.230676 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 11:54:37.251501 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.251473 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.251616 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.251524 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.251616 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.251552 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-web-config\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.251616 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.251598 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21691d59-0100-4c22-862f-ef85e52cc843-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.251727 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.251653 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.251727 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.251681 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.251727 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.251720 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.251813 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.251783 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.251861 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.251841 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.251926 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.251876 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-config\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.251974 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.251946 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.252027 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.251974 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.252027 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.252000 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.252126 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.252027 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21691d59-0100-4c22-862f-ef85e52cc843-config-out\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.252126 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.252054 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.252126 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.252084 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.252304 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.252131 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.252304 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.252222 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.252304 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.252290 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.252452 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.252321 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74x9g\" (UniqueName: \"kubernetes.io/projected/21691d59-0100-4c22-862f-ef85e52cc843-kube-api-access-74x9g\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.304884 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.304839 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.103751978 podStartE2EDuration="6.304825286s" totalImagesPullingTime="3.188882125s" totalInitContainerRuntime="12.191183ms" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:37.304373805 +0000 UTC m=+30.414216710" watchObservedRunningTime="2026-10-30 11:54:37.304825286 +0000 UTC m=+30.414668204" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.353583 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.353651 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.353683 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74x9g\" (UniqueName: \"kubernetes.io/projected/21691d59-0100-4c22-862f-ef85e52cc843-kube-api-access-74x9g\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.353737 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.353804 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.353830 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-web-config\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.353874 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21691d59-0100-4c22-862f-ef85e52cc843-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.353938 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.353964 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.353992 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.354043 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.354077 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.354104 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-config\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.354132 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.354162 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.354186 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.354211 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21691d59-0100-4c22-862f-ef85e52cc843-config-out\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.354236 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.354291 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.354324 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.355739 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.357295 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.356676 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.359325 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.359276 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.360933 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.360905 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.361880 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.361855 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.363418 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.362880 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.363418 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.363378 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.365471 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.365437 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.366153 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.366108 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-web-config\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.366233 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.366146 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21691d59-0100-4c22-862f-ef85e52cc843-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.367724 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.367680 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.369019 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.368803 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21691d59-0100-4c22-862f-ef85e52cc843-config-out\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.369019 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.368932 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.369244 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.369219 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-config\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.369682 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.369662 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.370184 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.370146 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.370291 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.370197 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.370556 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.370510 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.372465 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.372445 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.387918 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.387891 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74x9g\" (UniqueName: \"kubernetes.io/projected/21691d59-0100-4c22-862f-ef85e52cc843-kube-api-access-74x9g\") pod \"prometheus-k8s-0\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.524965 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.524938 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:37.857664 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.857552 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" podStartSLOduration=2.849144784 podStartE2EDuration="5.857533058s" totalImagesPullingTime="3.008388274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:37.356945535 +0000 UTC m=+30.466788417" watchObservedRunningTime="2026-10-30 11:54:37.857533058 +0000 UTC m=+30.967375940" Oct 30 11:54:37.858288 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:37.858266 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 11:54:37.861603 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:54:37.861577 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21691d59_0100_4c22_862f_ef85e52cc843.slice/crio-60031573209abe33c3949d54dd42f2a1b075662a765946b9c1d94ee7a1f2fe78 WatchSource:0}: Error finding container 60031573209abe33c3949d54dd42f2a1b075662a765946b9c1d94ee7a1f2fe78: Status 404 returned error can't find the container with id 60031573209abe33c3949d54dd42f2a1b075662a765946b9c1d94ee7a1f2fe78 Oct 30 11:54:38.168923 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:38.168895 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:38.211195 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:38.211157 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65459c448-trpdm" event={"ID":"2f8e0b61-2169-4d89-8315-4092541ac795","Type":"ContainerStarted","Data":"dbc0619f5d218545526e526c9204e8138099871b6d202d8168c12fc2e12c47c9"} Oct 30 11:54:38.211663 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:38.211230 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65459c448-trpdm" event={"ID":"2f8e0b61-2169-4d89-8315-4092541ac795","Type":"ContainerStarted","Data":"ae32e75d26d0176dbfa00508cc5c95b3fbc2ba1a26b3b1a84d7f0eff08706500"} Oct 30 11:54:38.211663 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:38.211283 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65459c448-trpdm" event={"ID":"2f8e0b61-2169-4d89-8315-4092541ac795","Type":"ContainerStarted","Data":"7ece179a388358eaf6bf4ec726e2b969db45bb428a1f7f904004496569e21284"} Oct 30 11:54:38.213111 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:38.213074 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqhdc" event={"ID":"59d12821-d129-4883-89c6-219a359e3143","Type":"ContainerStarted","Data":"ccfa8f0cd3a1356893a8afc56e1a6492d4a2d6e33f268ec427bde5271f5d1495"} Oct 30 11:54:38.213211 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:38.213132 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqhdc" Oct 30 11:54:38.214863 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:38.214842 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" event={"ID":"f122aa99-4d33-450f-acd4-da132dd1cb2e","Type":"ContainerStarted","Data":"20094c67d2989d3a7d011423049594d7e3c757c33fce3618b89e1fd9777d4c24"} Oct 30 11:54:38.216642 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:38.216617 4413 generic.go:356] "Generic (PLEG): container finished" podID="21691d59-0100-4c22-862f-ef85e52cc843" containerID="3ec281234e8ea0d533ae6a915eddcb2b2ac1dcc6f0907bca239da1b4b72cd6ac" exitCode=0 Oct 30 11:54:38.216758 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:38.216694 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21691d59-0100-4c22-862f-ef85e52cc843","Type":"ContainerDied","Data":"3ec281234e8ea0d533ae6a915eddcb2b2ac1dcc6f0907bca239da1b4b72cd6ac"} Oct 30 11:54:38.216819 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:38.216758 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21691d59-0100-4c22-862f-ef85e52cc843","Type":"ContainerStarted","Data":"60031573209abe33c3949d54dd42f2a1b075662a765946b9c1d94ee7a1f2fe78"} Oct 30 11:54:38.219570 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:38.219552 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqhdc" Oct 30 11:54:38.258497 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:38.258455 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-65459c448-trpdm" podStartSLOduration=1.447613502 podStartE2EDuration="3.258444975s" totalImagesPullingTime="1.810831473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:38.25809766 +0000 UTC m=+31.367940604" watchObservedRunningTime="2026-10-30 11:54:38.258444975 +0000 UTC m=+31.368287850" Oct 30 11:54:38.287463 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:38.287431 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:38.362742 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:38.362661 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqhdc" podStartSLOduration=1.888735638 podStartE2EDuration="3.362648333s" totalImagesPullingTime="1.473912695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:38.361845454 +0000 UTC m=+31.471688334" watchObservedRunningTime="2026-10-30 11:54:38.362648333 +0000 UTC m=+31.472491208" Oct 30 11:54:38.404693 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:38.404650 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" podStartSLOduration=2.839646424 podStartE2EDuration="4.404637604s" totalImagesPullingTime="1.56499118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:38.404204764 +0000 UTC m=+31.514047646" watchObservedRunningTime="2026-10-30 11:54:38.404637604 +0000 UTC m=+31.514480484" Oct 30 11:54:39.527134 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:39.527093 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:39.744472 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:39.744436 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:39.893006 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:39.892966 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:40.023896 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:40.023862 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:40.185592 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:40.185517 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:41.644825 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:41.644794 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:41.761722 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:41.761693 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:42.236426 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:42.236382 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21691d59-0100-4c22-862f-ef85e52cc843","Type":"ContainerStarted","Data":"02bc038c191c0264ceb3dacb164abec92ecc92ef6dfb2499806c2008a2e5ab0a"} Oct 30 11:54:42.236546 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:42.236518 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21691d59-0100-4c22-862f-ef85e52cc843","Type":"ContainerStarted","Data":"13f62071326c0a0d5842016e339135381eb24533df8ff593e4065c98e3646791"} Oct 30 11:54:42.236589 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:42.236563 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21691d59-0100-4c22-862f-ef85e52cc843","Type":"ContainerStarted","Data":"f77f530e876afffceeb633231f7137324b639d2b5dbaf28b933b88b684018ba9"} Oct 30 11:54:42.236625 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:42.236596 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21691d59-0100-4c22-862f-ef85e52cc843","Type":"ContainerStarted","Data":"89586a55ae2bb4d090cb5289ab347ff2cbe9e42cdb72e36720e46da4b9f59fd4"} Oct 30 11:54:42.236669 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:42.236636 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21691d59-0100-4c22-862f-ef85e52cc843","Type":"ContainerStarted","Data":"474981c55e68958876739a479493a25dcd7f586bc771fc72083d6965b2931564"} Oct 30 11:54:43.243447 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:43.243416 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21691d59-0100-4c22-862f-ef85e52cc843","Type":"ContainerStarted","Data":"baa1d6b941ccfd5cf847917a78d24642cf9d455506817dafecf0585d8097f073"} Oct 30 11:54:43.336910 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:43.336839 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.861327219 podStartE2EDuration="6.336825156s" totalImagesPullingTime="3.463480237s" totalInitContainerRuntime="12.0177ms" isStatefulPod=false podCreationTimestamp="2026-10-30 11:54:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:54:43.324575023 +0000 UTC m=+36.434417904" watchObservedRunningTime="2026-10-30 11:54:43.336825156 +0000 UTC m=+36.446668069" Oct 30 11:54:44.227311 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:44.227284 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8676c985b4-fm9dg" Oct 30 11:54:45.470516 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:45.470481 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:45.604883 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:45.604852 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:46.221303 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:46.221267 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:46.338715 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:46.338667 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:47.525902 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:47.525864 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:54:48.734419 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:48.734373 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:48.859478 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:48.859432 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:51.338069 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:51.338015 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:51.451950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:51.451898 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:53.540319 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:53.540283 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:53.697309 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:53.697280 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:55.268535 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:55.268505 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:55.268921 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:55.268590 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:54:56.859201 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:56.859172 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:54:57.017790 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:54:57.017756 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:00.474150 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:00.474118 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:00.595988 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:00.595955 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:01.929861 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:01.929817 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:02.082173 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:02.082136 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:03.646559 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:03.646529 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:03.665900 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:03.665878 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:04.335993 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:04.335961 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:05.814628 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:55:05.814588 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 11:55:09.050545 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:09.050507 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:09.168896 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:09.168860 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:10.825203 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:10.825163 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:10.953951 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:10.953919 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:11.697468 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:11.697432 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:11.826954 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:11.826920 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:11.965720 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:11.965634 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:12.259151 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:12.259064 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:12.395102 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:12.395069 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:15.274081 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:15.274054 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:55:15.277797 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:15.277778 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-94cc96cf4-mq2jm" Oct 30 11:55:15.495215 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:15.495180 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:15.608665 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:15.608584 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:16.236058 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:16.236030 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:16.347691 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:16.347655 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:17.248333 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:17.248303 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:17.404082 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:17.404049 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:18.243204 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:18.243174 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:18.357230 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:18.357200 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:21.283481 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:21.283452 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 11:55:21.283912 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:21.283777 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="alertmanager" containerID="cri-o://d9e715992c746509bcb7a6172e3dd55b1b3623ce16ba0cc3d55095bba24488d7" gracePeriod=120 Oct 30 11:55:21.283912 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:21.283808 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="kube-rbac-proxy-metric" containerID="cri-o://516002f8c38f4c394a1dae1fade992bb05187f42544eb44b5b536f46988a1cb1" gracePeriod=120 Oct 30 11:55:21.283912 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:21.283849 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="prom-label-proxy" containerID="cri-o://3099befc61e1d442366a385f55a50d1f9ee2940162d918fcc8d1a84b93b7ac45" gracePeriod=120 Oct 30 11:55:21.284101 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:21.283883 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="kube-rbac-proxy" containerID="cri-o://e65896f9acc7585c182199cbb1dc2baee5489d22fa00b5b783b819626a7f2a51" gracePeriod=120 Oct 30 11:55:21.284101 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:21.283905 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="config-reloader" containerID="cri-o://ce3be1474d3a55b658527f293c4fb8baec9abeaff0234d0bb7326f881f7c0e2b" gracePeriod=120 Oct 30 11:55:21.284101 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:21.283865 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="kube-rbac-proxy-web" containerID="cri-o://f83251e509d39fcf471e1612945ba49ccd466243c0695496158457e7e1f0058c" gracePeriod=120 Oct 30 11:55:22.391659 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.391626 4413 generic.go:356] "Generic (PLEG): container finished" podID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerID="3099befc61e1d442366a385f55a50d1f9ee2940162d918fcc8d1a84b93b7ac45" exitCode=0 Oct 30 11:55:22.391659 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.391660 4413 generic.go:356] "Generic (PLEG): container finished" podID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerID="516002f8c38f4c394a1dae1fade992bb05187f42544eb44b5b536f46988a1cb1" exitCode=0 Oct 30 11:55:22.391998 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.391671 4413 generic.go:356] "Generic (PLEG): container finished" podID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerID="e65896f9acc7585c182199cbb1dc2baee5489d22fa00b5b783b819626a7f2a51" exitCode=0 Oct 30 11:55:22.391998 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.391680 4413 generic.go:356] "Generic (PLEG): container finished" podID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerID="ce3be1474d3a55b658527f293c4fb8baec9abeaff0234d0bb7326f881f7c0e2b" exitCode=0 Oct 30 11:55:22.391998 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.391689 4413 generic.go:356] "Generic (PLEG): container finished" podID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerID="d9e715992c746509bcb7a6172e3dd55b1b3623ce16ba0cc3d55095bba24488d7" exitCode=0 Oct 30 11:55:22.391998 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.391687 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0fc60ce-3b21-4f0c-84f0-58f627675180","Type":"ContainerDied","Data":"3099befc61e1d442366a385f55a50d1f9ee2940162d918fcc8d1a84b93b7ac45"} Oct 30 11:55:22.391998 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.391750 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0fc60ce-3b21-4f0c-84f0-58f627675180","Type":"ContainerDied","Data":"516002f8c38f4c394a1dae1fade992bb05187f42544eb44b5b536f46988a1cb1"} Oct 30 11:55:22.391998 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.391776 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0fc60ce-3b21-4f0c-84f0-58f627675180","Type":"ContainerDied","Data":"e65896f9acc7585c182199cbb1dc2baee5489d22fa00b5b783b819626a7f2a51"} Oct 30 11:55:22.391998 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.391800 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0fc60ce-3b21-4f0c-84f0-58f627675180","Type":"ContainerDied","Data":"ce3be1474d3a55b658527f293c4fb8baec9abeaff0234d0bb7326f881f7c0e2b"} Oct 30 11:55:22.391998 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.391824 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0fc60ce-3b21-4f0c-84f0-58f627675180","Type":"ContainerDied","Data":"d9e715992c746509bcb7a6172e3dd55b1b3623ce16ba0cc3d55095bba24488d7"} Oct 30 11:55:22.526839 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.526817 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:22.565337 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.565306 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0fc60ce-3b21-4f0c-84f0-58f627675180-tls-assets\" (UniqueName: \"kubernetes.io/projected/d0fc60ce-3b21-4f0c-84f0-58f627675180-tls-assets\") pod \"d0fc60ce-3b21-4f0c-84f0-58f627675180\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " Oct 30 11:55:22.565337 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.565340 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0fc60ce-3b21-4f0c-84f0-58f627675180-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0fc60ce-3b21-4f0c-84f0-58f627675180-alertmanager-main-db\") pod \"d0fc60ce-3b21-4f0c-84f0-58f627675180\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " Oct 30 11:55:22.565530 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.565365 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0fc60ce-3b21-4f0c-84f0-58f627675180-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0fc60ce-3b21-4f0c-84f0-58f627675180-alertmanager-trusted-ca-bundle\") pod \"d0fc60ce-3b21-4f0c-84f0-58f627675180\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " Oct 30 11:55:22.565530 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.565391 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0fc60ce-3b21-4f0c-84f0-58f627675180-kube-api-access-qn84j\" (UniqueName: \"kubernetes.io/projected/d0fc60ce-3b21-4f0c-84f0-58f627675180-kube-api-access-qn84j\") pod \"d0fc60ce-3b21-4f0c-84f0-58f627675180\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " Oct 30 11:55:22.565530 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.565505 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0fc60ce-3b21-4f0c-84f0-58f627675180-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0fc60ce-3b21-4f0c-84f0-58f627675180-metrics-client-ca\") pod \"d0fc60ce-3b21-4f0c-84f0-58f627675180\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " Oct 30 11:55:22.565666 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.565537 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0fc60ce-3b21-4f0c-84f0-58f627675180-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0fc60ce-3b21-4f0c-84f0-58f627675180-config-out\") pod \"d0fc60ce-3b21-4f0c-84f0-58f627675180\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " Oct 30 11:55:22.565666 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.565571 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy-web\") pod \"d0fc60ce-3b21-4f0c-84f0-58f627675180\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " Oct 30 11:55:22.565666 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.565611 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy\") pod \"d0fc60ce-3b21-4f0c-84f0-58f627675180\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " Oct 30 11:55:22.565666 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.565642 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d0fc60ce-3b21-4f0c-84f0-58f627675180\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " Oct 30 11:55:22.565873 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.565685 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-web-config\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-web-config\") pod \"d0fc60ce-3b21-4f0c-84f0-58f627675180\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " Oct 30 11:55:22.565873 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.565712 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-cluster-tls-config\") pod \"d0fc60ce-3b21-4f0c-84f0-58f627675180\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " Oct 30 11:55:22.565873 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.565748 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-main-tls\") pod \"d0fc60ce-3b21-4f0c-84f0-58f627675180\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " Oct 30 11:55:22.565873 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.565731 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fc60ce-3b21-4f0c-84f0-58f627675180-alertmanager-main-db" pod "d0fc60ce-3b21-4f0c-84f0-58f627675180" (UID: "d0fc60ce-3b21-4f0c-84f0-58f627675180"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:55:22.565873 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.565789 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-config-volume\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-config-volume\") pod \"d0fc60ce-3b21-4f0c-84f0-58f627675180\" (UID: \"d0fc60ce-3b21-4f0c-84f0-58f627675180\") " Oct 30 11:55:22.565873 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.565826 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0fc60ce-3b21-4f0c-84f0-58f627675180-alertmanager-trusted-ca-bundle" pod "d0fc60ce-3b21-4f0c-84f0-58f627675180" (UID: "d0fc60ce-3b21-4f0c-84f0-58f627675180"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:22.566481 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.566130 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0fc60ce-3b21-4f0c-84f0-58f627675180-metrics-client-ca" pod "d0fc60ce-3b21-4f0c-84f0-58f627675180" (UID: "d0fc60ce-3b21-4f0c-84f0-58f627675180"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:22.566481 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.566300 4413 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0fc60ce-3b21-4f0c-84f0-58f627675180-alertmanager-main-db\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:22.566481 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.566322 4413 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0fc60ce-3b21-4f0c-84f0-58f627675180-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:22.568553 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.568525 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0fc60ce-3b21-4f0c-84f0-58f627675180-kube-api-access-qn84j" pod "d0fc60ce-3b21-4f0c-84f0-58f627675180" (UID: "d0fc60ce-3b21-4f0c-84f0-58f627675180"). InnerVolumeSpecName "kube-api-access-qn84j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:55:22.568827 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.568551 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-config-volume" pod "d0fc60ce-3b21-4f0c-84f0-58f627675180" (UID: "d0fc60ce-3b21-4f0c-84f0-58f627675180"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:22.568996 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.568965 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy-web" pod "d0fc60ce-3b21-4f0c-84f0-58f627675180" (UID: "d0fc60ce-3b21-4f0c-84f0-58f627675180"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:22.569105 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.569000 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy" pod "d0fc60ce-3b21-4f0c-84f0-58f627675180" (UID: "d0fc60ce-3b21-4f0c-84f0-58f627675180"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:22.569105 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.569063 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fc60ce-3b21-4f0c-84f0-58f627675180-config-out" pod "d0fc60ce-3b21-4f0c-84f0-58f627675180" (UID: "d0fc60ce-3b21-4f0c-84f0-58f627675180"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:55:22.569353 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.569332 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0fc60ce-3b21-4f0c-84f0-58f627675180-tls-assets" pod "d0fc60ce-3b21-4f0c-84f0-58f627675180" (UID: "d0fc60ce-3b21-4f0c-84f0-58f627675180"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:55:22.570487 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.570470 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy-metric" pod "d0fc60ce-3b21-4f0c-84f0-58f627675180" (UID: "d0fc60ce-3b21-4f0c-84f0-58f627675180"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:22.570742 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.570722 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-main-tls" pod "d0fc60ce-3b21-4f0c-84f0-58f627675180" (UID: "d0fc60ce-3b21-4f0c-84f0-58f627675180"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:22.573705 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.573611 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-cluster-tls-config" pod "d0fc60ce-3b21-4f0c-84f0-58f627675180" (UID: "d0fc60ce-3b21-4f0c-84f0-58f627675180"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:22.580653 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.580559 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-web-config" pod "d0fc60ce-3b21-4f0c-84f0-58f627675180" (UID: "d0fc60ce-3b21-4f0c-84f0-58f627675180"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:22.667401 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.667320 4413 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:22.667401 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.667346 4413 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:22.667401 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.667356 4413 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:22.667401 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.667366 4413 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-web-config\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:22.667401 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.667376 4413 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-cluster-tls-config\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:22.667401 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.667386 4413 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-secret-alertmanager-main-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:22.667401 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.667397 4413 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0fc60ce-3b21-4f0c-84f0-58f627675180-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:22.667861 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.667412 4413 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0fc60ce-3b21-4f0c-84f0-58f627675180-tls-assets\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:22.667861 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.667422 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn84j\" (UniqueName: \"kubernetes.io/projected/d0fc60ce-3b21-4f0c-84f0-58f627675180-kube-api-access-qn84j\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:22.667861 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.667431 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0fc60ce-3b21-4f0c-84f0-58f627675180-metrics-client-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:22.667861 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:22.667439 4413 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0fc60ce-3b21-4f0c-84f0-58f627675180-config-out\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:23.404079 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.404041 4413 generic.go:356] "Generic (PLEG): container finished" podID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerID="f83251e509d39fcf471e1612945ba49ccd466243c0695496158457e7e1f0058c" exitCode=0 Oct 30 11:55:23.404469 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.404121 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0fc60ce-3b21-4f0c-84f0-58f627675180","Type":"ContainerDied","Data":"f83251e509d39fcf471e1612945ba49ccd466243c0695496158457e7e1f0058c"} Oct 30 11:55:23.404469 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.404145 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.404469 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.404218 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0fc60ce-3b21-4f0c-84f0-58f627675180","Type":"ContainerDied","Data":"1e6a5270aa2b6b9c2a7eae78e643a021e2e7136a799a28ace303980ee4fc963b"} Oct 30 11:55:23.404469 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.404271 4413 scope.go:122] "RemoveContainer" containerID="3099befc61e1d442366a385f55a50d1f9ee2940162d918fcc8d1a84b93b7ac45" Oct 30 11:55:23.412705 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.412644 4413 scope.go:122] "RemoveContainer" containerID="516002f8c38f4c394a1dae1fade992bb05187f42544eb44b5b536f46988a1cb1" Oct 30 11:55:23.419649 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.419632 4413 scope.go:122] "RemoveContainer" containerID="e65896f9acc7585c182199cbb1dc2baee5489d22fa00b5b783b819626a7f2a51" Oct 30 11:55:23.426585 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.426562 4413 scope.go:122] "RemoveContainer" containerID="f83251e509d39fcf471e1612945ba49ccd466243c0695496158457e7e1f0058c" Oct 30 11:55:23.433768 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.433749 4413 scope.go:122] "RemoveContainer" containerID="ce3be1474d3a55b658527f293c4fb8baec9abeaff0234d0bb7326f881f7c0e2b" Oct 30 11:55:23.440662 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.440643 4413 scope.go:122] "RemoveContainer" containerID="d9e715992c746509bcb7a6172e3dd55b1b3623ce16ba0cc3d55095bba24488d7" Oct 30 11:55:23.447399 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.447384 4413 scope.go:122] "RemoveContainer" containerID="a37438c3128d51a22760328efd1dc02f86cfdc26a9203c7a2d0b4312438d7862" Oct 30 11:55:23.454269 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.454234 4413 scope.go:122] "RemoveContainer" containerID="3099befc61e1d442366a385f55a50d1f9ee2940162d918fcc8d1a84b93b7ac45" Oct 30 11:55:23.454542 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:55:23.454522 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3099befc61e1d442366a385f55a50d1f9ee2940162d918fcc8d1a84b93b7ac45\": container with ID starting with 3099befc61e1d442366a385f55a50d1f9ee2940162d918fcc8d1a84b93b7ac45 not found: ID does not exist" containerID="3099befc61e1d442366a385f55a50d1f9ee2940162d918fcc8d1a84b93b7ac45" Oct 30 11:55:23.454601 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.454551 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3099befc61e1d442366a385f55a50d1f9ee2940162d918fcc8d1a84b93b7ac45"} err="failed to get container status \"3099befc61e1d442366a385f55a50d1f9ee2940162d918fcc8d1a84b93b7ac45\": rpc error: code = NotFound desc = could not find container \"3099befc61e1d442366a385f55a50d1f9ee2940162d918fcc8d1a84b93b7ac45\": container with ID starting with 3099befc61e1d442366a385f55a50d1f9ee2940162d918fcc8d1a84b93b7ac45 not found: ID does not exist" Oct 30 11:55:23.454601 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.454584 4413 scope.go:122] "RemoveContainer" containerID="516002f8c38f4c394a1dae1fade992bb05187f42544eb44b5b536f46988a1cb1" Oct 30 11:55:23.454832 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:55:23.454818 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516002f8c38f4c394a1dae1fade992bb05187f42544eb44b5b536f46988a1cb1\": container with ID starting with 516002f8c38f4c394a1dae1fade992bb05187f42544eb44b5b536f46988a1cb1 not found: ID does not exist" containerID="516002f8c38f4c394a1dae1fade992bb05187f42544eb44b5b536f46988a1cb1" Oct 30 11:55:23.454876 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.454835 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516002f8c38f4c394a1dae1fade992bb05187f42544eb44b5b536f46988a1cb1"} err="failed to get container status \"516002f8c38f4c394a1dae1fade992bb05187f42544eb44b5b536f46988a1cb1\": rpc error: code = NotFound desc = could not find container \"516002f8c38f4c394a1dae1fade992bb05187f42544eb44b5b536f46988a1cb1\": container with ID starting with 516002f8c38f4c394a1dae1fade992bb05187f42544eb44b5b536f46988a1cb1 not found: ID does not exist" Oct 30 11:55:23.454876 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.454847 4413 scope.go:122] "RemoveContainer" containerID="e65896f9acc7585c182199cbb1dc2baee5489d22fa00b5b783b819626a7f2a51" Oct 30 11:55:23.455072 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:55:23.455053 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65896f9acc7585c182199cbb1dc2baee5489d22fa00b5b783b819626a7f2a51\": container with ID starting with e65896f9acc7585c182199cbb1dc2baee5489d22fa00b5b783b819626a7f2a51 not found: ID does not exist" containerID="e65896f9acc7585c182199cbb1dc2baee5489d22fa00b5b783b819626a7f2a51" Oct 30 11:55:23.455111 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.455079 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65896f9acc7585c182199cbb1dc2baee5489d22fa00b5b783b819626a7f2a51"} err="failed to get container status \"e65896f9acc7585c182199cbb1dc2baee5489d22fa00b5b783b819626a7f2a51\": rpc error: code = NotFound desc = could not find container \"e65896f9acc7585c182199cbb1dc2baee5489d22fa00b5b783b819626a7f2a51\": container with ID starting with e65896f9acc7585c182199cbb1dc2baee5489d22fa00b5b783b819626a7f2a51 not found: ID does not exist" Oct 30 11:55:23.455111 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.455097 4413 scope.go:122] "RemoveContainer" containerID="f83251e509d39fcf471e1612945ba49ccd466243c0695496158457e7e1f0058c" Oct 30 11:55:23.455348 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:55:23.455333 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83251e509d39fcf471e1612945ba49ccd466243c0695496158457e7e1f0058c\": container with ID starting with f83251e509d39fcf471e1612945ba49ccd466243c0695496158457e7e1f0058c not found: ID does not exist" containerID="f83251e509d39fcf471e1612945ba49ccd466243c0695496158457e7e1f0058c" Oct 30 11:55:23.455390 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.455352 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83251e509d39fcf471e1612945ba49ccd466243c0695496158457e7e1f0058c"} err="failed to get container status \"f83251e509d39fcf471e1612945ba49ccd466243c0695496158457e7e1f0058c\": rpc error: code = NotFound desc = could not find container \"f83251e509d39fcf471e1612945ba49ccd466243c0695496158457e7e1f0058c\": container with ID starting with f83251e509d39fcf471e1612945ba49ccd466243c0695496158457e7e1f0058c not found: ID does not exist" Oct 30 11:55:23.455390 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.455363 4413 scope.go:122] "RemoveContainer" containerID="ce3be1474d3a55b658527f293c4fb8baec9abeaff0234d0bb7326f881f7c0e2b" Oct 30 11:55:23.455565 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:55:23.455550 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce3be1474d3a55b658527f293c4fb8baec9abeaff0234d0bb7326f881f7c0e2b\": container with ID starting with ce3be1474d3a55b658527f293c4fb8baec9abeaff0234d0bb7326f881f7c0e2b not found: ID does not exist" containerID="ce3be1474d3a55b658527f293c4fb8baec9abeaff0234d0bb7326f881f7c0e2b" Oct 30 11:55:23.455605 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.455568 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce3be1474d3a55b658527f293c4fb8baec9abeaff0234d0bb7326f881f7c0e2b"} err="failed to get container status \"ce3be1474d3a55b658527f293c4fb8baec9abeaff0234d0bb7326f881f7c0e2b\": rpc error: code = NotFound desc = could not find container \"ce3be1474d3a55b658527f293c4fb8baec9abeaff0234d0bb7326f881f7c0e2b\": container with ID starting with ce3be1474d3a55b658527f293c4fb8baec9abeaff0234d0bb7326f881f7c0e2b not found: ID does not exist" Oct 30 11:55:23.455605 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.455581 4413 scope.go:122] "RemoveContainer" containerID="d9e715992c746509bcb7a6172e3dd55b1b3623ce16ba0cc3d55095bba24488d7" Oct 30 11:55:23.455784 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:55:23.455769 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9e715992c746509bcb7a6172e3dd55b1b3623ce16ba0cc3d55095bba24488d7\": container with ID starting with d9e715992c746509bcb7a6172e3dd55b1b3623ce16ba0cc3d55095bba24488d7 not found: ID does not exist" containerID="d9e715992c746509bcb7a6172e3dd55b1b3623ce16ba0cc3d55095bba24488d7" Oct 30 11:55:23.455821 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.455787 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e715992c746509bcb7a6172e3dd55b1b3623ce16ba0cc3d55095bba24488d7"} err="failed to get container status \"d9e715992c746509bcb7a6172e3dd55b1b3623ce16ba0cc3d55095bba24488d7\": rpc error: code = NotFound desc = could not find container \"d9e715992c746509bcb7a6172e3dd55b1b3623ce16ba0cc3d55095bba24488d7\": container with ID starting with d9e715992c746509bcb7a6172e3dd55b1b3623ce16ba0cc3d55095bba24488d7 not found: ID does not exist" Oct 30 11:55:23.455821 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.455799 4413 scope.go:122] "RemoveContainer" containerID="a37438c3128d51a22760328efd1dc02f86cfdc26a9203c7a2d0b4312438d7862" Oct 30 11:55:23.455986 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:55:23.455970 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a37438c3128d51a22760328efd1dc02f86cfdc26a9203c7a2d0b4312438d7862\": container with ID starting with a37438c3128d51a22760328efd1dc02f86cfdc26a9203c7a2d0b4312438d7862 not found: ID does not exist" containerID="a37438c3128d51a22760328efd1dc02f86cfdc26a9203c7a2d0b4312438d7862" Oct 30 11:55:23.456023 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.455989 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a37438c3128d51a22760328efd1dc02f86cfdc26a9203c7a2d0b4312438d7862"} err="failed to get container status \"a37438c3128d51a22760328efd1dc02f86cfdc26a9203c7a2d0b4312438d7862\": rpc error: code = NotFound desc = could not find container \"a37438c3128d51a22760328efd1dc02f86cfdc26a9203c7a2d0b4312438d7862\": container with ID starting with a37438c3128d51a22760328efd1dc02f86cfdc26a9203c7a2d0b4312438d7862 not found: ID does not exist" Oct 30 11:55:23.469215 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.469186 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 11:55:23.481242 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.481218 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 11:55:23.505900 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.505875 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 11:55:23.506160 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506149 4413 cpu_manager.go:414] "containerMap: removing container" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="alertmanager" Oct 30 11:55:23.506201 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506161 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="alertmanager" Oct 30 11:55:23.506201 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506171 4413 cpu_manager.go:414] "containerMap: removing container" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="kube-rbac-proxy-metric" Oct 30 11:55:23.506201 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506176 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="kube-rbac-proxy-metric" Oct 30 11:55:23.506201 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506188 4413 cpu_manager.go:414] "containerMap: removing container" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="kube-rbac-proxy-web" Oct 30 11:55:23.506201 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506193 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="kube-rbac-proxy-web" Oct 30 11:55:23.506201 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506199 4413 cpu_manager.go:414] "containerMap: removing container" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="kube-rbac-proxy" Oct 30 11:55:23.506201 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506204 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="kube-rbac-proxy" Oct 30 11:55:23.506459 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506212 4413 cpu_manager.go:414] "containerMap: removing container" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="prom-label-proxy" Oct 30 11:55:23.506459 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506217 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="prom-label-proxy" Oct 30 11:55:23.506459 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506228 4413 cpu_manager.go:414] "containerMap: removing container" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="config-reloader" Oct 30 11:55:23.506459 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506232 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="config-reloader" Oct 30 11:55:23.506459 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506261 4413 cpu_manager.go:414] "containerMap: removing container" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="init-config-reloader" Oct 30 11:55:23.506459 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506266 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="init-config-reloader" Oct 30 11:55:23.506459 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506313 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="alertmanager" Oct 30 11:55:23.506459 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506321 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="kube-rbac-proxy-metric" Oct 30 11:55:23.506459 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506328 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="kube-rbac-proxy-web" Oct 30 11:55:23.506459 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506334 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="kube-rbac-proxy" Oct 30 11:55:23.506459 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506340 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="prom-label-proxy" Oct 30 11:55:23.506459 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.506346 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" containerName="config-reloader" Oct 30 11:55:23.510116 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.510098 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.525549 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.525527 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 11:55:23.525650 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.525636 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 11:55:23.525792 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.525767 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 11:55:23.526066 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.526051 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 11:55:23.526148 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.526099 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m2ldd\"" Oct 30 11:55:23.526405 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.526388 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 11:55:23.526480 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.526396 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 11:55:23.537633 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.537501 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 11:55:23.537633 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.537570 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 11:55:23.537819 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.537748 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 11:55:23.539221 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.539198 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 11:55:23.573746 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.573704 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/148b3790-0c62-4d7f-be84-be2bdc04595c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.573746 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.573747 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/148b3790-0c62-4d7f-be84-be2bdc04595c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.574008 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.573771 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.574008 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.573887 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.574008 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.573918 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.574008 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.573934 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/148b3790-0c62-4d7f-be84-be2bdc04595c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.574008 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.573983 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/148b3790-0c62-4d7f-be84-be2bdc04595c-config-out\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.574189 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.574036 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.574189 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.574055 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp55r\" (UniqueName: \"kubernetes.io/projected/148b3790-0c62-4d7f-be84-be2bdc04595c-kube-api-access-qp55r\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.574189 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.574081 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.574189 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.574103 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-config-volume\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.574189 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.574122 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/148b3790-0c62-4d7f-be84-be2bdc04595c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.574189 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.574165 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-web-config\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.675105 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.675007 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-config-volume\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.675105 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.675067 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/148b3790-0c62-4d7f-be84-be2bdc04595c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.675105 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.675096 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-web-config\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.675409 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.675127 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/148b3790-0c62-4d7f-be84-be2bdc04595c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.675409 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.675155 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/148b3790-0c62-4d7f-be84-be2bdc04595c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.675409 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.675188 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.675409 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.675233 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.675409 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.675288 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.675745 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.675717 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/148b3790-0c62-4d7f-be84-be2bdc04595c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.675832 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.675784 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/148b3790-0c62-4d7f-be84-be2bdc04595c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.675832 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.675821 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/148b3790-0c62-4d7f-be84-be2bdc04595c-config-out\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.675932 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.675856 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.675932 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.675889 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp55r\" (UniqueName: \"kubernetes.io/projected/148b3790-0c62-4d7f-be84-be2bdc04595c-kube-api-access-qp55r\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.676032 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.675951 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.676862 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.676569 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/148b3790-0c62-4d7f-be84-be2bdc04595c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.677838 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.677791 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/148b3790-0c62-4d7f-be84-be2bdc04595c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.678550 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.678523 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-config-volume\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.678932 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.678898 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/148b3790-0c62-4d7f-be84-be2bdc04595c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.679075 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.679054 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.679141 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.679054 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.679524 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.679500 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.679524 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.679519 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-web-config\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.679677 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.679507 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.679677 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.679520 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/148b3790-0c62-4d7f-be84-be2bdc04595c-config-out\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.680681 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.680665 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/148b3790-0c62-4d7f-be84-be2bdc04595c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.700151 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.700127 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp55r\" (UniqueName: \"kubernetes.io/projected/148b3790-0c62-4d7f-be84-be2bdc04595c-kube-api-access-qp55r\") pod \"alertmanager-main-0\" (UID: \"148b3790-0c62-4d7f-be84-be2bdc04595c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:23.826818 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:23.826785 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 11:55:24.158989 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:24.158948 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 11:55:24.161600 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:55:24.161573 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148b3790_0c62_4d7f_be84_be2bdc04595c.slice/crio-6e72ace2d3098231c6835cd2b6405aa93cc9bdb2a20f2ec21d1dd1447bb856f6 WatchSource:0}: Error finding container 6e72ace2d3098231c6835cd2b6405aa93cc9bdb2a20f2ec21d1dd1447bb856f6: Status 404 returned error can't find the container with id 6e72ace2d3098231c6835cd2b6405aa93cc9bdb2a20f2ec21d1dd1447bb856f6 Oct 30 11:55:24.409902 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:24.409826 4413 generic.go:356] "Generic (PLEG): container finished" podID="148b3790-0c62-4d7f-be84-be2bdc04595c" containerID="6172cd2db2cd53f65c2fd2ae31744b259bf12cadf53f4e90875a0137a9097371" exitCode=0 Oct 30 11:55:24.409902 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:24.409859 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"148b3790-0c62-4d7f-be84-be2bdc04595c","Type":"ContainerDied","Data":"6172cd2db2cd53f65c2fd2ae31744b259bf12cadf53f4e90875a0137a9097371"} Oct 30 11:55:24.410275 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:24.409907 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"148b3790-0c62-4d7f-be84-be2bdc04595c","Type":"ContainerStarted","Data":"6e72ace2d3098231c6835cd2b6405aa93cc9bdb2a20f2ec21d1dd1447bb856f6"} Oct 30 11:55:25.016518 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:25.016437 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0fc60ce-3b21-4f0c-84f0-58f627675180" path="/var/lib/kubelet/pods/d0fc60ce-3b21-4f0c-84f0-58f627675180/volumes" Oct 30 11:55:25.416527 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:25.416496 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"148b3790-0c62-4d7f-be84-be2bdc04595c","Type":"ContainerStarted","Data":"9bbd36fbf096b4d37f25e3f3ba13a38bb6057c99f81676a23b2b0466e6031d96"} Oct 30 11:55:25.416924 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:25.416556 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"148b3790-0c62-4d7f-be84-be2bdc04595c","Type":"ContainerStarted","Data":"3f12f684830dbbeb015ee0823d11d80309369dbb55373c073f04ce4bfe04b54f"} Oct 30 11:55:25.416924 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:25.416581 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"148b3790-0c62-4d7f-be84-be2bdc04595c","Type":"ContainerStarted","Data":"8af0e34eb0f96ff4fbaa0d17b0cd65810b1109d51b1e59ee125fcf9537d9e2df"} Oct 30 11:55:25.416924 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:25.416604 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"148b3790-0c62-4d7f-be84-be2bdc04595c","Type":"ContainerStarted","Data":"d7c94f024554cc359e97d99d1c35a1208ea1e5fb2707b8e3747d136634621a38"} Oct 30 11:55:25.416924 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:25.416627 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"148b3790-0c62-4d7f-be84-be2bdc04595c","Type":"ContainerStarted","Data":"7e276ff7796948ed49184eef3f005a383691021b2c3394c836ba023f4c950ecb"} Oct 30 11:55:25.416924 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:25.416649 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"148b3790-0c62-4d7f-be84-be2bdc04595c","Type":"ContainerStarted","Data":"a9b75f91902eb7b0f95e028ffe54217a1c5f363bdbf4a0965f43dd234371fd39"} Oct 30 11:55:25.507065 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:25.507007 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.496044977 podStartE2EDuration="2.506994495s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.949518ms" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:25.506551271 +0000 UTC m=+78.616394151" watchObservedRunningTime="2026-10-30 11:55:25.506994495 +0000 UTC m=+78.616837376" Oct 30 11:55:25.831101 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:25.831020 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 11:55:25.831361 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:25.831340 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="prometheus" containerID="cri-o://474981c55e68958876739a479493a25dcd7f586bc771fc72083d6965b2931564" gracePeriod=600 Oct 30 11:55:25.831432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:25.831375 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="thanos-sidecar" containerID="cri-o://f77f530e876afffceeb633231f7137324b639d2b5dbaf28b933b88b684018ba9" gracePeriod=600 Oct 30 11:55:25.831432 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:25.831395 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="config-reloader" containerID="cri-o://89586a55ae2bb4d090cb5289ab347ff2cbe9e42cdb72e36720e46da4b9f59fd4" gracePeriod=600 Oct 30 11:55:25.831543 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:25.831464 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="kube-rbac-proxy-web" containerID="cri-o://13f62071326c0a0d5842016e339135381eb24533df8ff593e4065c98e3646791" gracePeriod=600 Oct 30 11:55:25.831543 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:25.831502 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="kube-rbac-proxy" containerID="cri-o://02bc038c191c0264ceb3dacb164abec92ecc92ef6dfb2499806c2008a2e5ab0a" gracePeriod=600 Oct 30 11:55:25.831645 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:25.831413 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="kube-rbac-proxy-thanos" containerID="cri-o://baa1d6b941ccfd5cf847917a78d24642cf9d455506817dafecf0585d8097f073" gracePeriod=600 Oct 30 11:55:26.165382 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:26.165350 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:26.279073 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:26.279041 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:26.424044 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:26.423963 4413 generic.go:356] "Generic (PLEG): container finished" podID="21691d59-0100-4c22-862f-ef85e52cc843" containerID="baa1d6b941ccfd5cf847917a78d24642cf9d455506817dafecf0585d8097f073" exitCode=0 Oct 30 11:55:26.424044 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:26.423990 4413 generic.go:356] "Generic (PLEG): container finished" podID="21691d59-0100-4c22-862f-ef85e52cc843" containerID="02bc038c191c0264ceb3dacb164abec92ecc92ef6dfb2499806c2008a2e5ab0a" exitCode=0 Oct 30 11:55:26.424044 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:26.423996 4413 generic.go:356] "Generic (PLEG): container finished" podID="21691d59-0100-4c22-862f-ef85e52cc843" containerID="f77f530e876afffceeb633231f7137324b639d2b5dbaf28b933b88b684018ba9" exitCode=0 Oct 30 11:55:26.424044 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:26.424004 4413 generic.go:356] "Generic (PLEG): container finished" podID="21691d59-0100-4c22-862f-ef85e52cc843" containerID="89586a55ae2bb4d090cb5289ab347ff2cbe9e42cdb72e36720e46da4b9f59fd4" exitCode=0 Oct 30 11:55:26.424044 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:26.424009 4413 generic.go:356] "Generic (PLEG): container finished" podID="21691d59-0100-4c22-862f-ef85e52cc843" containerID="474981c55e68958876739a479493a25dcd7f586bc771fc72083d6965b2931564" exitCode=0 Oct 30 11:55:26.424044 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:26.424017 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21691d59-0100-4c22-862f-ef85e52cc843","Type":"ContainerDied","Data":"baa1d6b941ccfd5cf847917a78d24642cf9d455506817dafecf0585d8097f073"} Oct 30 11:55:26.424599 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:26.424075 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21691d59-0100-4c22-862f-ef85e52cc843","Type":"ContainerDied","Data":"02bc038c191c0264ceb3dacb164abec92ecc92ef6dfb2499806c2008a2e5ab0a"} Oct 30 11:55:26.424599 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:26.424104 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21691d59-0100-4c22-862f-ef85e52cc843","Type":"ContainerDied","Data":"f77f530e876afffceeb633231f7137324b639d2b5dbaf28b933b88b684018ba9"} Oct 30 11:55:26.424599 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:26.424128 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21691d59-0100-4c22-862f-ef85e52cc843","Type":"ContainerDied","Data":"89586a55ae2bb4d090cb5289ab347ff2cbe9e42cdb72e36720e46da4b9f59fd4"} Oct 30 11:55:26.424599 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:26.424150 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21691d59-0100-4c22-862f-ef85e52cc843","Type":"ContainerDied","Data":"474981c55e68958876739a479493a25dcd7f586bc771fc72083d6965b2931564"} Oct 30 11:55:27.078530 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.078505 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.104207 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104180 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-thanos-prometheus-http-client-file\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104376 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104235 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104376 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104289 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-metrics-client-ca\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104376 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104310 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-config\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-config\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104376 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104345 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21691d59-0100-4c22-862f-ef85e52cc843-kube-api-access-74x9g\" (UniqueName: \"kubernetes.io/projected/21691d59-0100-4c22-862f-ef85e52cc843-kube-api-access-74x9g\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104570 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104377 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-tls\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104570 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104404 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-0\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104570 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104440 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-trusted-ca-bundle\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104570 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104469 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104570 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104493 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-db\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104570 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104525 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-grpc-tls\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104570 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104550 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-1\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104937 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104590 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21691d59-0100-4c22-862f-ef85e52cc843-tls-assets\" (UniqueName: \"kubernetes.io/projected/21691d59-0100-4c22-862f-ef85e52cc843-tls-assets\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104937 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104621 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-kube-rbac-proxy\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104937 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104678 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-kubelet-serving-ca-bundle\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104937 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104708 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-2\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104937 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104773 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21691d59-0100-4c22-862f-ef85e52cc843-config-out\" (UniqueName: \"kubernetes.io/empty-dir/21691d59-0100-4c22-862f-ef85e52cc843-config-out\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104937 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104805 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-metrics-client-certs\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104937 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104837 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-serving-certs-ca-bundle\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.104937 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104869 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-web-config\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-web-config\") pod \"21691d59-0100-4c22-862f-ef85e52cc843\" (UID: \"21691d59-0100-4c22-862f-ef85e52cc843\") " Oct 30 11:55:27.105562 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.104951 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-trusted-ca-bundle" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:27.105562 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.105012 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-metrics-client-ca" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:27.105562 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.105160 4413 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-metrics-client-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.105562 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.105180 4413 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.106778 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.106745 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-0" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:27.107645 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.107365 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-grpc-tls" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:27.108605 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.107898 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-kubelet-serving-ca-bundle" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:27.108605 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.108445 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-thanos-sidecar-tls" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:27.108932 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.108715 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-serving-certs-ca-bundle" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:27.109135 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.109030 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-tls" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:27.109327 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.109287 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-db" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:55:27.109417 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.109318 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-1" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:27.109520 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.109477 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-2" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 11:55:27.109808 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.109758 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-thanos-prometheus-http-client-file" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:27.110057 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.110035 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21691d59-0100-4c22-862f-ef85e52cc843-kube-api-access-74x9g" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "kube-api-access-74x9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:55:27.110468 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.110293 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-config" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:27.110468 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.110451 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21691d59-0100-4c22-862f-ef85e52cc843-config-out" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 11:55:27.110957 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.110933 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-metrics-client-certs" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:27.111382 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.111358 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21691d59-0100-4c22-862f-ef85e52cc843-tls-assets" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 11:55:27.111473 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.111377 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-kube-rbac-proxy-web" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:27.111810 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.111786 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-kube-rbac-proxy" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:27.123178 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.123158 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-web-config" pod "21691d59-0100-4c22-862f-ef85e52cc843" (UID: "21691d59-0100-4c22-862f-ef85e52cc843"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 11:55:27.205657 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.205625 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74x9g\" (UniqueName: \"kubernetes.io/projected/21691d59-0100-4c22-862f-ef85e52cc843-kube-api-access-74x9g\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.205657 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.205651 4413 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.205657 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.205661 4413 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.205950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.205670 4413 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.205950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.205679 4413 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-db\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.205950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.205688 4413 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-grpc-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.205950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.205696 4413 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.205950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.205703 4413 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21691d59-0100-4c22-862f-ef85e52cc843-tls-assets\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.205950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.205711 4413 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-kube-rbac-proxy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.205950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.205727 4413 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.205950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.205736 4413 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.205950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.205743 4413 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21691d59-0100-4c22-862f-ef85e52cc843-config-out\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.205950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.205754 4413 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-metrics-client-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.205950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.205762 4413 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21691d59-0100-4c22-862f-ef85e52cc843-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.205950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.205771 4413 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-web-config\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.205950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.205778 4413 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.205950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.205786 4413 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.205950 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.205794 4413 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/21691d59-0100-4c22-862f-ef85e52cc843-config\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 11:55:27.431167 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.431138 4413 generic.go:356] "Generic (PLEG): container finished" podID="21691d59-0100-4c22-862f-ef85e52cc843" containerID="13f62071326c0a0d5842016e339135381eb24533df8ff593e4065c98e3646791" exitCode=0 Oct 30 11:55:27.431586 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.431218 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.431586 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.431231 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21691d59-0100-4c22-862f-ef85e52cc843","Type":"ContainerDied","Data":"13f62071326c0a0d5842016e339135381eb24533df8ff593e4065c98e3646791"} Oct 30 11:55:27.431586 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.431314 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21691d59-0100-4c22-862f-ef85e52cc843","Type":"ContainerDied","Data":"60031573209abe33c3949d54dd42f2a1b075662a765946b9c1d94ee7a1f2fe78"} Oct 30 11:55:27.431586 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.431349 4413 scope.go:122] "RemoveContainer" containerID="baa1d6b941ccfd5cf847917a78d24642cf9d455506817dafecf0585d8097f073" Oct 30 11:55:27.439222 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.439201 4413 scope.go:122] "RemoveContainer" containerID="02bc038c191c0264ceb3dacb164abec92ecc92ef6dfb2499806c2008a2e5ab0a" Oct 30 11:55:27.446122 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.446104 4413 scope.go:122] "RemoveContainer" containerID="13f62071326c0a0d5842016e339135381eb24533df8ff593e4065c98e3646791" Oct 30 11:55:27.452816 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.452798 4413 scope.go:122] "RemoveContainer" containerID="f77f530e876afffceeb633231f7137324b639d2b5dbaf28b933b88b684018ba9" Oct 30 11:55:27.459356 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.459337 4413 scope.go:122] "RemoveContainer" containerID="89586a55ae2bb4d090cb5289ab347ff2cbe9e42cdb72e36720e46da4b9f59fd4" Oct 30 11:55:27.466050 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.466036 4413 scope.go:122] "RemoveContainer" containerID="474981c55e68958876739a479493a25dcd7f586bc771fc72083d6965b2931564" Oct 30 11:55:27.472906 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.472889 4413 scope.go:122] "RemoveContainer" containerID="3ec281234e8ea0d533ae6a915eddcb2b2ac1dcc6f0907bca239da1b4b72cd6ac" Oct 30 11:55:27.479454 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.479438 4413 scope.go:122] "RemoveContainer" containerID="baa1d6b941ccfd5cf847917a78d24642cf9d455506817dafecf0585d8097f073" Oct 30 11:55:27.479701 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:55:27.479682 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa1d6b941ccfd5cf847917a78d24642cf9d455506817dafecf0585d8097f073\": container with ID starting with baa1d6b941ccfd5cf847917a78d24642cf9d455506817dafecf0585d8097f073 not found: ID does not exist" containerID="baa1d6b941ccfd5cf847917a78d24642cf9d455506817dafecf0585d8097f073" Oct 30 11:55:27.479742 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.479710 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa1d6b941ccfd5cf847917a78d24642cf9d455506817dafecf0585d8097f073"} err="failed to get container status \"baa1d6b941ccfd5cf847917a78d24642cf9d455506817dafecf0585d8097f073\": rpc error: code = NotFound desc = could not find container \"baa1d6b941ccfd5cf847917a78d24642cf9d455506817dafecf0585d8097f073\": container with ID starting with baa1d6b941ccfd5cf847917a78d24642cf9d455506817dafecf0585d8097f073 not found: ID does not exist" Oct 30 11:55:27.479742 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.479727 4413 scope.go:122] "RemoveContainer" containerID="02bc038c191c0264ceb3dacb164abec92ecc92ef6dfb2499806c2008a2e5ab0a" Oct 30 11:55:27.479951 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:55:27.479935 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02bc038c191c0264ceb3dacb164abec92ecc92ef6dfb2499806c2008a2e5ab0a\": container with ID starting with 02bc038c191c0264ceb3dacb164abec92ecc92ef6dfb2499806c2008a2e5ab0a not found: ID does not exist" containerID="02bc038c191c0264ceb3dacb164abec92ecc92ef6dfb2499806c2008a2e5ab0a" Oct 30 11:55:27.479995 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.479955 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02bc038c191c0264ceb3dacb164abec92ecc92ef6dfb2499806c2008a2e5ab0a"} err="failed to get container status \"02bc038c191c0264ceb3dacb164abec92ecc92ef6dfb2499806c2008a2e5ab0a\": rpc error: code = NotFound desc = could not find container \"02bc038c191c0264ceb3dacb164abec92ecc92ef6dfb2499806c2008a2e5ab0a\": container with ID starting with 02bc038c191c0264ceb3dacb164abec92ecc92ef6dfb2499806c2008a2e5ab0a not found: ID does not exist" Oct 30 11:55:27.479995 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.479967 4413 scope.go:122] "RemoveContainer" containerID="13f62071326c0a0d5842016e339135381eb24533df8ff593e4065c98e3646791" Oct 30 11:55:27.480180 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:55:27.480162 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f62071326c0a0d5842016e339135381eb24533df8ff593e4065c98e3646791\": container with ID starting with 13f62071326c0a0d5842016e339135381eb24533df8ff593e4065c98e3646791 not found: ID does not exist" containerID="13f62071326c0a0d5842016e339135381eb24533df8ff593e4065c98e3646791" Oct 30 11:55:27.480227 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.480189 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f62071326c0a0d5842016e339135381eb24533df8ff593e4065c98e3646791"} err="failed to get container status \"13f62071326c0a0d5842016e339135381eb24533df8ff593e4065c98e3646791\": rpc error: code = NotFound desc = could not find container \"13f62071326c0a0d5842016e339135381eb24533df8ff593e4065c98e3646791\": container with ID starting with 13f62071326c0a0d5842016e339135381eb24533df8ff593e4065c98e3646791 not found: ID does not exist" Oct 30 11:55:27.480227 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.480206 4413 scope.go:122] "RemoveContainer" containerID="f77f530e876afffceeb633231f7137324b639d2b5dbaf28b933b88b684018ba9" Oct 30 11:55:27.480449 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:55:27.480434 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f77f530e876afffceeb633231f7137324b639d2b5dbaf28b933b88b684018ba9\": container with ID starting with f77f530e876afffceeb633231f7137324b639d2b5dbaf28b933b88b684018ba9 not found: ID does not exist" containerID="f77f530e876afffceeb633231f7137324b639d2b5dbaf28b933b88b684018ba9" Oct 30 11:55:27.480494 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.480452 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77f530e876afffceeb633231f7137324b639d2b5dbaf28b933b88b684018ba9"} err="failed to get container status \"f77f530e876afffceeb633231f7137324b639d2b5dbaf28b933b88b684018ba9\": rpc error: code = NotFound desc = could not find container \"f77f530e876afffceeb633231f7137324b639d2b5dbaf28b933b88b684018ba9\": container with ID starting with f77f530e876afffceeb633231f7137324b639d2b5dbaf28b933b88b684018ba9 not found: ID does not exist" Oct 30 11:55:27.480494 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.480463 4413 scope.go:122] "RemoveContainer" containerID="89586a55ae2bb4d090cb5289ab347ff2cbe9e42cdb72e36720e46da4b9f59fd4" Oct 30 11:55:27.480708 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:55:27.480690 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89586a55ae2bb4d090cb5289ab347ff2cbe9e42cdb72e36720e46da4b9f59fd4\": container with ID starting with 89586a55ae2bb4d090cb5289ab347ff2cbe9e42cdb72e36720e46da4b9f59fd4 not found: ID does not exist" containerID="89586a55ae2bb4d090cb5289ab347ff2cbe9e42cdb72e36720e46da4b9f59fd4" Oct 30 11:55:27.480761 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.480710 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89586a55ae2bb4d090cb5289ab347ff2cbe9e42cdb72e36720e46da4b9f59fd4"} err="failed to get container status \"89586a55ae2bb4d090cb5289ab347ff2cbe9e42cdb72e36720e46da4b9f59fd4\": rpc error: code = NotFound desc = could not find container \"89586a55ae2bb4d090cb5289ab347ff2cbe9e42cdb72e36720e46da4b9f59fd4\": container with ID starting with 89586a55ae2bb4d090cb5289ab347ff2cbe9e42cdb72e36720e46da4b9f59fd4 not found: ID does not exist" Oct 30 11:55:27.480761 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.480722 4413 scope.go:122] "RemoveContainer" containerID="474981c55e68958876739a479493a25dcd7f586bc771fc72083d6965b2931564" Oct 30 11:55:27.480900 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:55:27.480886 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474981c55e68958876739a479493a25dcd7f586bc771fc72083d6965b2931564\": container with ID starting with 474981c55e68958876739a479493a25dcd7f586bc771fc72083d6965b2931564 not found: ID does not exist" containerID="474981c55e68958876739a479493a25dcd7f586bc771fc72083d6965b2931564" Oct 30 11:55:27.480938 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.480903 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474981c55e68958876739a479493a25dcd7f586bc771fc72083d6965b2931564"} err="failed to get container status \"474981c55e68958876739a479493a25dcd7f586bc771fc72083d6965b2931564\": rpc error: code = NotFound desc = could not find container \"474981c55e68958876739a479493a25dcd7f586bc771fc72083d6965b2931564\": container with ID starting with 474981c55e68958876739a479493a25dcd7f586bc771fc72083d6965b2931564 not found: ID does not exist" Oct 30 11:55:27.480938 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.480914 4413 scope.go:122] "RemoveContainer" containerID="3ec281234e8ea0d533ae6a915eddcb2b2ac1dcc6f0907bca239da1b4b72cd6ac" Oct 30 11:55:27.481135 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:55:27.481120 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ec281234e8ea0d533ae6a915eddcb2b2ac1dcc6f0907bca239da1b4b72cd6ac\": container with ID starting with 3ec281234e8ea0d533ae6a915eddcb2b2ac1dcc6f0907bca239da1b4b72cd6ac not found: ID does not exist" containerID="3ec281234e8ea0d533ae6a915eddcb2b2ac1dcc6f0907bca239da1b4b72cd6ac" Oct 30 11:55:27.481175 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.481136 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ec281234e8ea0d533ae6a915eddcb2b2ac1dcc6f0907bca239da1b4b72cd6ac"} err="failed to get container status \"3ec281234e8ea0d533ae6a915eddcb2b2ac1dcc6f0907bca239da1b4b72cd6ac\": rpc error: code = NotFound desc = could not find container \"3ec281234e8ea0d533ae6a915eddcb2b2ac1dcc6f0907bca239da1b4b72cd6ac\": container with ID starting with 3ec281234e8ea0d533ae6a915eddcb2b2ac1dcc6f0907bca239da1b4b72cd6ac not found: ID does not exist" Oct 30 11:55:27.485362 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.485343 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 11:55:27.497005 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.496977 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 11:55:27.533068 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533041 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 11:55:27.533349 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533336 4413 cpu_manager.go:414] "containerMap: removing container" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="kube-rbac-proxy-web" Oct 30 11:55:27.533349 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533349 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="kube-rbac-proxy-web" Oct 30 11:55:27.533447 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533363 4413 cpu_manager.go:414] "containerMap: removing container" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="thanos-sidecar" Oct 30 11:55:27.533447 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533368 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="thanos-sidecar" Oct 30 11:55:27.533447 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533376 4413 cpu_manager.go:414] "containerMap: removing container" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="kube-rbac-proxy-thanos" Oct 30 11:55:27.533447 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533381 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="kube-rbac-proxy-thanos" Oct 30 11:55:27.533447 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533390 4413 cpu_manager.go:414] "containerMap: removing container" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="init-config-reloader" Oct 30 11:55:27.533447 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533394 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="init-config-reloader" Oct 30 11:55:27.533447 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533410 4413 cpu_manager.go:414] "containerMap: removing container" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="kube-rbac-proxy" Oct 30 11:55:27.533447 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533415 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="kube-rbac-proxy" Oct 30 11:55:27.533447 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533423 4413 cpu_manager.go:414] "containerMap: removing container" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="prometheus" Oct 30 11:55:27.533447 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533427 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="prometheus" Oct 30 11:55:27.533447 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533433 4413 cpu_manager.go:414] "containerMap: removing container" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="config-reloader" Oct 30 11:55:27.533447 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533437 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="config-reloader" Oct 30 11:55:27.534024 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533479 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="kube-rbac-proxy" Oct 30 11:55:27.534024 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533486 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="prometheus" Oct 30 11:55:27.534024 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533492 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="config-reloader" Oct 30 11:55:27.534024 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533499 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="kube-rbac-proxy-web" Oct 30 11:55:27.534024 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533505 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="thanos-sidecar" Oct 30 11:55:27.534024 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.533510 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="21691d59-0100-4c22-862f-ef85e52cc843" containerName="kube-rbac-proxy-thanos" Oct 30 11:55:27.540949 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.540921 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.556145 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.556119 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 11:55:27.556353 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.556172 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 11:55:27.556353 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.556232 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 11:55:27.556353 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.556261 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 11:55:27.556599 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.556373 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-l2pdp\"" Oct 30 11:55:27.556599 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.556395 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 11:55:27.556599 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.556401 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 11:55:27.556599 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.556428 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7dn3dgofmkf17\"" Oct 30 11:55:27.556599 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.556448 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 11:55:27.556599 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.556454 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 11:55:27.556599 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.556558 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 11:55:27.556847 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.556652 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 11:55:27.556847 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.556667 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 11:55:27.557219 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.557201 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 11:55:27.570123 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.570101 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 11:55:27.571030 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.571002 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 11:55:27.572170 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.572153 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 11:55:27.608700 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.608662 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.608700 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.608697 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xqbg\" (UniqueName: \"kubernetes.io/projected/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-kube-api-access-4xqbg\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.608917 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.608718 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.608917 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.608739 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-config\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.608917 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.608823 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.608917 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.608910 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.609057 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.608939 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-config-out\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.609057 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.608955 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.609057 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.608976 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.609057 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.608993 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.609177 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.609064 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.609177 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.609125 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.609177 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.609151 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-web-config\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.609177 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.609167 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.609341 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.609186 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.609341 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.609235 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.609341 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.609296 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.609341 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.609327 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.609476 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.609351 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.609476 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.609410 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.710115 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.710045 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.710115 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.710088 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-config-out\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.710115 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.710111 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.710395 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.710139 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.710395 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.710163 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.710395 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.710192 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.710395 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.710219 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.710395 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.710262 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-web-config\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.711074 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.710996 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.711205 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.711096 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.711205 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.711186 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.711335 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.711217 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.711335 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.711225 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.711335 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.711281 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.711335 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.711309 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.711546 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.711336 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.711546 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.711359 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.711546 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.711390 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.711546 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.711435 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.711546 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.711461 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xqbg\" (UniqueName: \"kubernetes.io/projected/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-kube-api-access-4xqbg\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.711546 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.711494 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.711546 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.711523 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-config\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.711546 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.711549 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.712108 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.711577 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.712108 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.712068 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.712342 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.712322 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.713609 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.713204 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-config-out\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.713609 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.713525 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-web-config\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.714104 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.714075 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.714483 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.714456 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.714691 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.714570 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.714691 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.714584 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.715153 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.715120 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.715706 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.715675 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.716185 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.716161 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.716391 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.716373 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.716667 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.716650 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.716780 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.716757 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.717082 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.717062 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-config\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.737212 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.737186 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xqbg\" (UniqueName: \"kubernetes.io/projected/06a5bd76-a68f-4c5a-a085-d3f1d91724d5-kube-api-access-4xqbg\") pod \"prometheus-k8s-0\" (UID: \"06a5bd76-a68f-4c5a-a085-d3f1d91724d5\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:27.858047 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:27.857994 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:28.190573 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:28.190541 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 11:55:28.193157 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:55:28.193122 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a5bd76_a68f_4c5a_a085_d3f1d91724d5.slice/crio-5d16c64685a53b6dcf3ebdec528c2d04e915f1f605ea12daae32aa1d56f7fb0f WatchSource:0}: Error finding container 5d16c64685a53b6dcf3ebdec528c2d04e915f1f605ea12daae32aa1d56f7fb0f: Status 404 returned error can't find the container with id 5d16c64685a53b6dcf3ebdec528c2d04e915f1f605ea12daae32aa1d56f7fb0f Oct 30 11:55:28.437337 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:28.437304 4413 generic.go:356] "Generic (PLEG): container finished" podID="06a5bd76-a68f-4c5a-a085-d3f1d91724d5" containerID="4a249f5342367c84cd808e6e5ae64f125935e1657f0581c0fb37ba588eb4ed26" exitCode=0 Oct 30 11:55:28.437693 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:28.437366 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a5bd76-a68f-4c5a-a085-d3f1d91724d5","Type":"ContainerDied","Data":"4a249f5342367c84cd808e6e5ae64f125935e1657f0581c0fb37ba588eb4ed26"} Oct 30 11:55:28.437693 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:28.437431 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a5bd76-a68f-4c5a-a085-d3f1d91724d5","Type":"ContainerStarted","Data":"5d16c64685a53b6dcf3ebdec528c2d04e915f1f605ea12daae32aa1d56f7fb0f"} Oct 30 11:55:29.020491 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:29.017699 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21691d59-0100-4c22-862f-ef85e52cc843" path="/var/lib/kubelet/pods/21691d59-0100-4c22-862f-ef85e52cc843/volumes" Oct 30 11:55:29.444120 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:29.444085 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a5bd76-a68f-4c5a-a085-d3f1d91724d5","Type":"ContainerStarted","Data":"ba9852810bcec6050df19fc4d1656d442e81d643346585ca734e4c9137f9fe60"} Oct 30 11:55:29.444611 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:29.444168 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a5bd76-a68f-4c5a-a085-d3f1d91724d5","Type":"ContainerStarted","Data":"30b7a319b39c183c5ef3dfc71a93976d3757dadbb72895bdc97626a6f3d2c57d"} Oct 30 11:55:29.444611 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:29.444195 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a5bd76-a68f-4c5a-a085-d3f1d91724d5","Type":"ContainerStarted","Data":"7a29042f47a3950cf56f2559d41d42df53ca5fc230750c987efb6055fa49b080"} Oct 30 11:55:29.444611 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:29.444217 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a5bd76-a68f-4c5a-a085-d3f1d91724d5","Type":"ContainerStarted","Data":"21bf6504b68df198f30c4f68937d249d1cb145cc64e1461485a6c4302315612d"} Oct 30 11:55:29.444611 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:29.444239 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a5bd76-a68f-4c5a-a085-d3f1d91724d5","Type":"ContainerStarted","Data":"f4ba2b0fb503f8588185adf1a6dcc53585cfe94fe4eda1b5c2b3c98fee124ecd"} Oct 30 11:55:29.444611 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:29.444279 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"06a5bd76-a68f-4c5a-a085-d3f1d91724d5","Type":"ContainerStarted","Data":"e8dc3d31c1e57dc955c4dfc99e97f87f9c05253828ad3238a41a740123de36f6"} Oct 30 11:55:29.511018 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:29.510968 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.49637228 podStartE2EDuration="2.51095541s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.58313ms" isStatefulPod=false podCreationTimestamp="2026-10-30 11:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:55:29.509870632 +0000 UTC m=+82.619713513" watchObservedRunningTime="2026-10-30 11:55:29.51095541 +0000 UTC m=+82.620798320" Oct 30 11:55:30.454566 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:30.454509 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:30.580220 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:30.580190 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:55:32.858828 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:32.858792 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:55:35.820578 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:55:35.820533 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 11:55:53.655351 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:55:53.655307 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:56:05.813517 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:56:05.813482 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 11:56:27.858712 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:56:27.858611 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:56:27.874675 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:56:27.874652 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:56:28.673983 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:56:28.673955 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 11:56:35.813310 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:56:35.813273 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 11:56:42.133299 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:56:42.133238 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:56:42.252803 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:56:42.252776 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:56:53.783682 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:56:53.783648 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:05.812655 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:57:05.812618 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 11:57:06.617408 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:06.617359 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:06.736470 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:06.736433 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:15.592980 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:15.592942 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:15.712340 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:15.712312 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:31.087151 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:31.087115 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:31.262800 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:31.262746 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:31.399129 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:31.399099 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:32.661685 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:32.661642 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:32.777862 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:32.777821 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:34.053932 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:34.053905 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:35.812605 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:57:35.812560 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 11:57:36.438960 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:36.438930 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:36.557385 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:36.557347 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:37.883821 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:37.883785 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:38.003804 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:38.003772 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:38.117495 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:38.117468 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:41.051355 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:41.051328 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:42.436123 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:42.436093 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:46.665879 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:46.665812 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:46.783644 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:46.783609 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:53.902300 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:53.902187 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:55.167590 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:55.167551 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:57:55.283621 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:57:55.283588 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:05.812002 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:58:05.811964 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 11:58:14.807015 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:14.806979 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:18.436288 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:18.436234 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:18.554424 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:18.554390 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:19.626862 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:19.626830 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:19.857447 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:19.857420 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86"] Oct 30 11:58:19.860764 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:19.860751 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:19.881466 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:19.881418 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86"] Oct 30 11:58:19.931545 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:19.931518 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdxnh\" (UniqueName: \"kubernetes.io/projected/b7693302-f5f2-4863-9173-1efbc18b77e2-kube-api-access-jdxnh\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-6wf86\" (UID: \"b7693302-f5f2-4863-9173-1efbc18b77e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:19.931545 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:19.931548 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b7693302-f5f2-4863-9173-1efbc18b77e2-manifests\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-6wf86\" (UID: \"b7693302-f5f2-4863-9173-1efbc18b77e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:19.931743 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:19.931575 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7693302-f5f2-4863-9173-1efbc18b77e2-tmp\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-6wf86\" (UID: \"b7693302-f5f2-4863-9173-1efbc18b77e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:19.931743 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:19.931595 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b7693302-f5f2-4863-9173-1efbc18b77e2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-6wf86\" (UID: \"b7693302-f5f2-4863-9173-1efbc18b77e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:19.931743 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:19.931687 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7693302-f5f2-4863-9173-1efbc18b77e2-webhook-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-6wf86\" (UID: \"b7693302-f5f2-4863-9173-1efbc18b77e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:19.931897 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:19.931796 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b7693302-f5f2-4863-9173-1efbc18b77e2-metrics-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-6wf86\" (UID: \"b7693302-f5f2-4863-9173-1efbc18b77e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:19.947333 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:19.947314 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 11:58:19.947500 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:19.947485 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 11:58:19.947560 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:19.947502 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 11:58:19.947600 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:19.947559 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-b7n4r\"" Oct 30 11:58:19.950903 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:19.950890 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 11:58:19.950961 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:19.950934 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 11:58:20.032318 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.032294 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdxnh\" (UniqueName: \"kubernetes.io/projected/b7693302-f5f2-4863-9173-1efbc18b77e2-kube-api-access-jdxnh\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-6wf86\" (UID: \"b7693302-f5f2-4863-9173-1efbc18b77e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:20.032318 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.032320 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b7693302-f5f2-4863-9173-1efbc18b77e2-manifests\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-6wf86\" (UID: \"b7693302-f5f2-4863-9173-1efbc18b77e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:20.032509 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.032349 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7693302-f5f2-4863-9173-1efbc18b77e2-tmp\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-6wf86\" (UID: \"b7693302-f5f2-4863-9173-1efbc18b77e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:20.032509 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.032366 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b7693302-f5f2-4863-9173-1efbc18b77e2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-6wf86\" (UID: \"b7693302-f5f2-4863-9173-1efbc18b77e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:20.032509 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.032388 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7693302-f5f2-4863-9173-1efbc18b77e2-webhook-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-6wf86\" (UID: \"b7693302-f5f2-4863-9173-1efbc18b77e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:20.032509 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.032423 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b7693302-f5f2-4863-9173-1efbc18b77e2-metrics-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-6wf86\" (UID: \"b7693302-f5f2-4863-9173-1efbc18b77e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:20.032737 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.032716 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b7693302-f5f2-4863-9173-1efbc18b77e2-manifests\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-6wf86\" (UID: \"b7693302-f5f2-4863-9173-1efbc18b77e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:20.034881 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.034856 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7693302-f5f2-4863-9173-1efbc18b77e2-tmp\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-6wf86\" (UID: \"b7693302-f5f2-4863-9173-1efbc18b77e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:20.034972 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.034936 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b7693302-f5f2-4863-9173-1efbc18b77e2-metrics-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-6wf86\" (UID: \"b7693302-f5f2-4863-9173-1efbc18b77e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:20.035029 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.035013 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b7693302-f5f2-4863-9173-1efbc18b77e2-webhook-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-6wf86\" (UID: \"b7693302-f5f2-4863-9173-1efbc18b77e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:20.035203 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.035184 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b7693302-f5f2-4863-9173-1efbc18b77e2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-6wf86\" (UID: \"b7693302-f5f2-4863-9173-1efbc18b77e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:20.058780 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.058762 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdxnh\" (UniqueName: \"kubernetes.io/projected/b7693302-f5f2-4863-9173-1efbc18b77e2-kube-api-access-jdxnh\") pod \"opendatahub-operator-controller-manager-64c9d5fd44-6wf86\" (UID: \"b7693302-f5f2-4863-9173-1efbc18b77e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:20.093038 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.093013 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:20.175385 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.175328 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:20.214546 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.214512 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:20.336491 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.336458 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:20.477243 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.477165 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:20.537435 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.537410 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86"] Oct 30 11:58:20.539603 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:58:20.539577 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7693302_f5f2_4863_9173_1efbc18b77e2.slice/crio-9641a6e4592485c95b0eeaa7782ad2f6962805dc79952b7c73a2aedf685917f2 WatchSource:0}: Error finding container 9641a6e4592485c95b0eeaa7782ad2f6962805dc79952b7c73a2aedf685917f2: Status 404 returned error can't find the container with id 9641a6e4592485c95b0eeaa7782ad2f6962805dc79952b7c73a2aedf685917f2 Oct 30 11:58:20.614938 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.614907 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:20.738722 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:20.738593 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:21.058424 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:21.058357 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" event={"ID":"b7693302-f5f2-4863-9173-1efbc18b77e2","Type":"ContainerStarted","Data":"9641a6e4592485c95b0eeaa7782ad2f6962805dc79952b7c73a2aedf685917f2"} Oct 30 11:58:22.021142 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:22.021108 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:22.240628 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:22.240592 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:22.635783 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:22.635754 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:22.755802 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:22.755772 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-vr2dm"] Oct 30 11:58:22.759521 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:22.759495 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vr2dm" Oct 30 11:58:22.793202 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:22.793170 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-vr2dm"] Oct 30 11:58:22.803878 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:22.803849 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 11:58:22.804030 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:22.803908 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 11:58:22.804030 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:22.803911 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-xj2fj\"" Oct 30 11:58:22.859929 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:22.859903 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7944\" (UniqueName: \"kubernetes.io/projected/7c623141-2d9f-412d-aea8-2a477bf96bec-kube-api-access-g7944\") pod \"jobset-operator-674cd59dc5-vr2dm\" (UID: \"7c623141-2d9f-412d-aea8-2a477bf96bec\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vr2dm" Oct 30 11:58:22.860096 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:22.859957 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c623141-2d9f-412d-aea8-2a477bf96bec-tmp\") pod \"jobset-operator-674cd59dc5-vr2dm\" (UID: \"7c623141-2d9f-412d-aea8-2a477bf96bec\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vr2dm" Oct 30 11:58:22.960952 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:22.960865 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c623141-2d9f-412d-aea8-2a477bf96bec-tmp\") pod \"jobset-operator-674cd59dc5-vr2dm\" (UID: \"7c623141-2d9f-412d-aea8-2a477bf96bec\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vr2dm" Oct 30 11:58:22.961108 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:22.960963 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7944\" (UniqueName: \"kubernetes.io/projected/7c623141-2d9f-412d-aea8-2a477bf96bec-kube-api-access-g7944\") pod \"jobset-operator-674cd59dc5-vr2dm\" (UID: \"7c623141-2d9f-412d-aea8-2a477bf96bec\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vr2dm" Oct 30 11:58:22.961264 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:22.961233 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c623141-2d9f-412d-aea8-2a477bf96bec-tmp\") pod \"jobset-operator-674cd59dc5-vr2dm\" (UID: \"7c623141-2d9f-412d-aea8-2a477bf96bec\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vr2dm" Oct 30 11:58:22.992069 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:22.992045 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7944\" (UniqueName: \"kubernetes.io/projected/7c623141-2d9f-412d-aea8-2a477bf96bec-kube-api-access-g7944\") pod \"jobset-operator-674cd59dc5-vr2dm\" (UID: \"7c623141-2d9f-412d-aea8-2a477bf96bec\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vr2dm" Oct 30 11:58:23.084151 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:23.084123 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vr2dm" Oct 30 11:58:23.404155 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:23.404088 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-vr2dm"] Oct 30 11:58:23.427810 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:58:23.424546 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c623141_2d9f_412d_aea8_2a477bf96bec.slice/crio-f23c305752dddfc16d0fec217c9bf885e6e3bf79675e0d70cb08636a5ea45f98 WatchSource:0}: Error finding container f23c305752dddfc16d0fec217c9bf885e6e3bf79675e0d70cb08636a5ea45f98: Status 404 returned error can't find the container with id f23c305752dddfc16d0fec217c9bf885e6e3bf79675e0d70cb08636a5ea45f98 Oct 30 11:58:24.071683 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:24.071651 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vr2dm" event={"ID":"7c623141-2d9f-412d-aea8-2a477bf96bec","Type":"ContainerStarted","Data":"f23c305752dddfc16d0fec217c9bf885e6e3bf79675e0d70cb08636a5ea45f98"} Oct 30 11:58:24.073223 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:24.073200 4413 generic.go:356] "Generic (PLEG): container finished" podID="b7693302-f5f2-4863-9173-1efbc18b77e2" containerID="f4e4249ea2d295feb70ea1b3851e485b126579ae71d32a7258c8fe84f897e0de" exitCode=0 Oct 30 11:58:24.073356 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:24.073285 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" event={"ID":"b7693302-f5f2-4863-9173-1efbc18b77e2","Type":"ContainerDied","Data":"f4e4249ea2d295feb70ea1b3851e485b126579ae71d32a7258c8fe84f897e0de"} Oct 30 11:58:24.252019 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:24.251921 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:24.616241 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:24.616155 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2vh57"] Oct 30 11:58:24.625912 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:24.625867 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vh57" Oct 30 11:58:24.639393 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:24.639364 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2vh57"] Oct 30 11:58:24.676177 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:24.676139 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76gsj\" (UniqueName: \"kubernetes.io/projected/5a000b39-dd68-485f-8832-c6a7c7b938be-kube-api-access-76gsj\") pod \"cert-manager-webhook-5dbbbf75cc-2vh57\" (UID: \"5a000b39-dd68-485f-8832-c6a7c7b938be\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vh57" Oct 30 11:58:24.676403 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:24.676185 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a000b39-dd68-485f-8832-c6a7c7b938be-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2vh57\" (UID: \"5a000b39-dd68-485f-8832-c6a7c7b938be\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vh57" Oct 30 11:58:24.680892 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:24.680703 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 11:58:24.680892 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:24.680855 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 11:58:24.680892 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:24.680855 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-7jh86\"" Oct 30 11:58:24.777284 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:24.777232 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76gsj\" (UniqueName: \"kubernetes.io/projected/5a000b39-dd68-485f-8832-c6a7c7b938be-kube-api-access-76gsj\") pod \"cert-manager-webhook-5dbbbf75cc-2vh57\" (UID: \"5a000b39-dd68-485f-8832-c6a7c7b938be\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vh57" Oct 30 11:58:24.777458 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:24.777317 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a000b39-dd68-485f-8832-c6a7c7b938be-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2vh57\" (UID: \"5a000b39-dd68-485f-8832-c6a7c7b938be\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vh57" Oct 30 11:58:24.804867 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:24.804836 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a000b39-dd68-485f-8832-c6a7c7b938be-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2vh57\" (UID: \"5a000b39-dd68-485f-8832-c6a7c7b938be\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vh57" Oct 30 11:58:24.805356 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:24.805329 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76gsj\" (UniqueName: \"kubernetes.io/projected/5a000b39-dd68-485f-8832-c6a7c7b938be-kube-api-access-76gsj\") pod \"cert-manager-webhook-5dbbbf75cc-2vh57\" (UID: \"5a000b39-dd68-485f-8832-c6a7c7b938be\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vh57" Oct 30 11:58:24.941504 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:24.941440 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vh57" Oct 30 11:58:25.083187 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:25.083151 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" event={"ID":"b7693302-f5f2-4863-9173-1efbc18b77e2","Type":"ContainerStarted","Data":"249661a28ec4e1ffa33c4d6591ccaea479f5c135e29edfba295dc52d16b9148e"} Oct 30 11:58:25.083590 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:25.083571 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:25.194017 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:25.193920 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" podStartSLOduration=2.642535526 podStartE2EDuration="6.193907256s" totalImagesPullingTime="3.145891043s" totalInitContainerRuntime="405.480687ms" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:25.190777998 +0000 UTC m=+258.300620905" watchObservedRunningTime="2026-10-30 11:58:25.193907256 +0000 UTC m=+258.303750137" Oct 30 11:58:25.257380 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:25.257346 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2vh57"] Oct 30 11:58:25.261504 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:58:25.261471 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a000b39_dd68_485f_8832_c6a7c7b938be.slice/crio-1e578f775b7d30d4f0aebe198b0b7ad3bdf796680e8b5201da1c7bb24b02882f WatchSource:0}: Error finding container 1e578f775b7d30d4f0aebe198b0b7ad3bdf796680e8b5201da1c7bb24b02882f: Status 404 returned error can't find the container with id 1e578f775b7d30d4f0aebe198b0b7ad3bdf796680e8b5201da1c7bb24b02882f Oct 30 11:58:25.322266 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:25.322217 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:26.089334 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:26.089296 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vh57" event={"ID":"5a000b39-dd68-485f-8832-c6a7c7b938be","Type":"ContainerStarted","Data":"1e578f775b7d30d4f0aebe198b0b7ad3bdf796680e8b5201da1c7bb24b02882f"} Oct 30 11:58:26.642037 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:26.641999 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:26.778316 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:26.778230 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:27.101810 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:27.101737 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vr2dm" event={"ID":"7c623141-2d9f-412d-aea8-2a477bf96bec","Type":"ContainerStarted","Data":"deecf368fc940a3df6efc6e8a5ac699d91568969ca5475b16b3057cc1ef24e26"} Oct 30 11:58:27.142542 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:27.142481 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vr2dm" podStartSLOduration=2.126842025 podStartE2EDuration="5.142461369s" totalImagesPullingTime="3.015619344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:27.141758824 +0000 UTC m=+260.251601708" watchObservedRunningTime="2026-10-30 11:58:27.142461369 +0000 UTC m=+260.252304255" Oct 30 11:58:29.112209 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:29.112124 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vh57" event={"ID":"5a000b39-dd68-485f-8832-c6a7c7b938be","Type":"ContainerStarted","Data":"fed83f53a6a036c4f9f5f973fb0d8b62231efd1eef2131d30de33c21a6b16957"} Oct 30 11:58:29.112608 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:29.112269 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vh57" Oct 30 11:58:29.152044 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:29.151995 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vh57" podStartSLOduration=1.602492889 podStartE2EDuration="5.151980064s" totalImagesPullingTime="3.549487175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:29.151792833 +0000 UTC m=+262.261635714" watchObservedRunningTime="2026-10-30 11:58:29.151980064 +0000 UTC m=+262.261822944" Oct 30 11:58:30.612204 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:30.612166 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:31.571499 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:31.571439 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:31.812439 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:31.812398 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:32.543701 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:32.543665 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-scmpv"] Oct 30 11:58:32.552099 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:32.552078 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-scmpv" Oct 30 11:58:32.565122 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:32.565101 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-scmpv"] Oct 30 11:58:32.576068 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:32.576050 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hkshd\"" Oct 30 11:58:32.651127 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:32.651080 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3483562-0d0a-46d4-956e-aadba630094c-bound-sa-token\") pod \"cert-manager-597b95cb84-scmpv\" (UID: \"b3483562-0d0a-46d4-956e-aadba630094c\") " pod="cert-manager/cert-manager-597b95cb84-scmpv" Oct 30 11:58:32.651351 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:32.651207 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf8d2\" (UniqueName: \"kubernetes.io/projected/b3483562-0d0a-46d4-956e-aadba630094c-kube-api-access-bf8d2\") pod \"cert-manager-597b95cb84-scmpv\" (UID: \"b3483562-0d0a-46d4-956e-aadba630094c\") " pod="cert-manager/cert-manager-597b95cb84-scmpv" Oct 30 11:58:32.751901 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:32.751874 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3483562-0d0a-46d4-956e-aadba630094c-bound-sa-token\") pod \"cert-manager-597b95cb84-scmpv\" (UID: \"b3483562-0d0a-46d4-956e-aadba630094c\") " pod="cert-manager/cert-manager-597b95cb84-scmpv" Oct 30 11:58:32.752061 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:32.751936 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf8d2\" (UniqueName: \"kubernetes.io/projected/b3483562-0d0a-46d4-956e-aadba630094c-kube-api-access-bf8d2\") pod \"cert-manager-597b95cb84-scmpv\" (UID: \"b3483562-0d0a-46d4-956e-aadba630094c\") " pod="cert-manager/cert-manager-597b95cb84-scmpv" Oct 30 11:58:32.774648 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:32.774618 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3483562-0d0a-46d4-956e-aadba630094c-bound-sa-token\") pod \"cert-manager-597b95cb84-scmpv\" (UID: \"b3483562-0d0a-46d4-956e-aadba630094c\") " pod="cert-manager/cert-manager-597b95cb84-scmpv" Oct 30 11:58:32.774849 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:32.774824 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf8d2\" (UniqueName: \"kubernetes.io/projected/b3483562-0d0a-46d4-956e-aadba630094c-kube-api-access-bf8d2\") pod \"cert-manager-597b95cb84-scmpv\" (UID: \"b3483562-0d0a-46d4-956e-aadba630094c\") " pod="cert-manager/cert-manager-597b95cb84-scmpv" Oct 30 11:58:32.864095 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:32.864028 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-scmpv" Oct 30 11:58:33.157723 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:33.157677 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-scmpv"] Oct 30 11:58:33.160635 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:58:33.160611 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3483562_0d0a_46d4_956e_aadba630094c.slice/crio-6f15c6ff3c39584f88db8d68b1f277200c99fa6ec43a758e3cde02383d9d2fd9 WatchSource:0}: Error finding container 6f15c6ff3c39584f88db8d68b1f277200c99fa6ec43a758e3cde02383d9d2fd9: Status 404 returned error can't find the container with id 6f15c6ff3c39584f88db8d68b1f277200c99fa6ec43a758e3cde02383d9d2fd9 Oct 30 11:58:34.133208 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:34.133172 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-scmpv" event={"ID":"b3483562-0d0a-46d4-956e-aadba630094c","Type":"ContainerStarted","Data":"df06522c9f1f03ae03d800e95104f0aa092546a29d964036716f737d65cb1f4f"} Oct 30 11:58:34.133208 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:34.133223 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-scmpv" event={"ID":"b3483562-0d0a-46d4-956e-aadba630094c","Type":"ContainerStarted","Data":"6f15c6ff3c39584f88db8d68b1f277200c99fa6ec43a758e3cde02383d9d2fd9"} Oct 30 11:58:34.172735 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:34.172679 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-scmpv" podStartSLOduration=2.17266598 podStartE2EDuration="2.17266598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:34.171907629 +0000 UTC m=+267.281750509" watchObservedRunningTime="2026-10-30 11:58:34.17266598 +0000 UTC m=+267.282508860" Oct 30 11:58:35.117685 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:35.117658 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vh57" Oct 30 11:58:35.812440 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:58:35.812393 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 11:58:36.092290 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:36.092201 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64c9d5fd44-6wf86" Oct 30 11:58:37.091834 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:37.091806 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:39.836362 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:39.836326 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:40.273494 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:40.273458 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:40.399602 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:40.399569 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:40.518640 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:40.518605 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:41.220858 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:41.220827 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:42.276486 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:42.276450 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:44.094683 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:44.094646 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:44.225045 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:44.225012 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:45.007026 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:45.006994 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:46.510734 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:46.510697 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:48.026559 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:48.026520 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:48.158594 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:48.158531 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:48.422401 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:48.422375 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:48.542367 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:48.542336 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:50.367645 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.367614 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-zrh8j"] Oct 30 11:58:50.369891 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.369876 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zrh8j" Oct 30 11:58:50.386443 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.386420 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 11:58:50.386594 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.386511 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jg5cf\"" Oct 30 11:58:50.392006 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.391984 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:50.398618 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.398543 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 11:58:50.398941 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.398922 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 11:58:50.399055 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.399034 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 11:58:50.400644 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.400619 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-zrh8j"] Oct 30 11:58:50.502410 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.502375 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87nlj\" (UniqueName: \"kubernetes.io/projected/6ec974e2-3c96-493d-b06a-f59ad7d60dce-kube-api-access-87nlj\") pod \"kuadrant-console-plugin-66c4fb575d-zrh8j\" (UID: \"6ec974e2-3c96-493d-b06a-f59ad7d60dce\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zrh8j" Oct 30 11:58:50.502598 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.502447 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ec974e2-3c96-493d-b06a-f59ad7d60dce-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-zrh8j\" (UID: \"6ec974e2-3c96-493d-b06a-f59ad7d60dce\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zrh8j" Oct 30 11:58:50.502598 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.502541 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6ec974e2-3c96-493d-b06a-f59ad7d60dce-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-zrh8j\" (UID: \"6ec974e2-3c96-493d-b06a-f59ad7d60dce\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zrh8j" Oct 30 11:58:50.551957 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.551919 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:50.603949 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.603924 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87nlj\" (UniqueName: \"kubernetes.io/projected/6ec974e2-3c96-493d-b06a-f59ad7d60dce-kube-api-access-87nlj\") pod \"kuadrant-console-plugin-66c4fb575d-zrh8j\" (UID: \"6ec974e2-3c96-493d-b06a-f59ad7d60dce\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zrh8j" Oct 30 11:58:50.604101 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.603970 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ec974e2-3c96-493d-b06a-f59ad7d60dce-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-zrh8j\" (UID: \"6ec974e2-3c96-493d-b06a-f59ad7d60dce\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zrh8j" Oct 30 11:58:50.604101 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.604001 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6ec974e2-3c96-493d-b06a-f59ad7d60dce-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-zrh8j\" (UID: \"6ec974e2-3c96-493d-b06a-f59ad7d60dce\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zrh8j" Oct 30 11:58:50.604692 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.604673 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6ec974e2-3c96-493d-b06a-f59ad7d60dce-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-zrh8j\" (UID: \"6ec974e2-3c96-493d-b06a-f59ad7d60dce\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zrh8j" Oct 30 11:58:50.606685 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.606656 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ec974e2-3c96-493d-b06a-f59ad7d60dce-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-zrh8j\" (UID: \"6ec974e2-3c96-493d-b06a-f59ad7d60dce\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zrh8j" Oct 30 11:58:50.626549 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.626522 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87nlj\" (UniqueName: \"kubernetes.io/projected/6ec974e2-3c96-493d-b06a-f59ad7d60dce-kube-api-access-87nlj\") pod \"kuadrant-console-plugin-66c4fb575d-zrh8j\" (UID: \"6ec974e2-3c96-493d-b06a-f59ad7d60dce\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zrh8j" Oct 30 11:58:50.682341 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.682319 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zrh8j" Oct 30 11:58:50.706894 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.706862 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:50.998699 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:50.998630 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-zrh8j"] Oct 30 11:58:51.001687 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:58:51.001654 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ec974e2_3c96_493d_b06a_f59ad7d60dce.slice/crio-e1eb299fc417b8384306a30ce0c14ef510c690bec533e059e45e8ea5a852291d WatchSource:0}: Error finding container e1eb299fc417b8384306a30ce0c14ef510c690bec533e059e45e8ea5a852291d: Status 404 returned error can't find the container with id e1eb299fc417b8384306a30ce0c14ef510c690bec533e059e45e8ea5a852291d Oct 30 11:58:51.169986 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.169953 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:51.200591 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.200563 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zrh8j" event={"ID":"6ec974e2-3c96-493d-b06a-f59ad7d60dce","Type":"ContainerStarted","Data":"e1eb299fc417b8384306a30ce0c14ef510c690bec533e059e45e8ea5a852291d"} Oct 30 11:58:51.336319 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.336223 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:51.337134 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.337110 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68fcc45f6c-twgp9"] Oct 30 11:58:51.340272 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.340240 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-twgp9" Oct 30 11:58:51.361974 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.361945 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 11:58:51.373098 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.373079 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 11:58:51.373421 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.373386 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 11:58:51.373468 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.373423 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 11:58:51.374215 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.374199 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-n7s7n\"" Oct 30 11:58:51.375570 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.375553 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68fcc45f6c-twgp9"] Oct 30 11:58:51.470728 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.470687 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:51.511953 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.511924 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eea523a-6a90-4f6b-9b89-c6c39bf6513f-tls-certs\") pod \"kube-auth-proxy-68fcc45f6c-twgp9\" (UID: \"7eea523a-6a90-4f6b-9b89-c6c39bf6513f\") " pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-twgp9" Oct 30 11:58:51.512131 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.511982 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49hk2\" (UniqueName: \"kubernetes.io/projected/7eea523a-6a90-4f6b-9b89-c6c39bf6513f-kube-api-access-49hk2\") pod \"kube-auth-proxy-68fcc45f6c-twgp9\" (UID: \"7eea523a-6a90-4f6b-9b89-c6c39bf6513f\") " pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-twgp9" Oct 30 11:58:51.512131 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.512116 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7eea523a-6a90-4f6b-9b89-c6c39bf6513f-tmp\") pod \"kube-auth-proxy-68fcc45f6c-twgp9\" (UID: \"7eea523a-6a90-4f6b-9b89-c6c39bf6513f\") " pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-twgp9" Oct 30 11:58:51.603084 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.603017 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:51.613263 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.613220 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7eea523a-6a90-4f6b-9b89-c6c39bf6513f-tmp\") pod \"kube-auth-proxy-68fcc45f6c-twgp9\" (UID: \"7eea523a-6a90-4f6b-9b89-c6c39bf6513f\") " pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-twgp9" Oct 30 11:58:51.613397 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.613305 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eea523a-6a90-4f6b-9b89-c6c39bf6513f-tls-certs\") pod \"kube-auth-proxy-68fcc45f6c-twgp9\" (UID: \"7eea523a-6a90-4f6b-9b89-c6c39bf6513f\") " pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-twgp9" Oct 30 11:58:51.613397 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.613341 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49hk2\" (UniqueName: \"kubernetes.io/projected/7eea523a-6a90-4f6b-9b89-c6c39bf6513f-kube-api-access-49hk2\") pod \"kube-auth-proxy-68fcc45f6c-twgp9\" (UID: \"7eea523a-6a90-4f6b-9b89-c6c39bf6513f\") " pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-twgp9" Oct 30 11:58:51.615561 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.615537 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7eea523a-6a90-4f6b-9b89-c6c39bf6513f-tmp\") pod \"kube-auth-proxy-68fcc45f6c-twgp9\" (UID: \"7eea523a-6a90-4f6b-9b89-c6c39bf6513f\") " pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-twgp9" Oct 30 11:58:51.615811 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.615794 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eea523a-6a90-4f6b-9b89-c6c39bf6513f-tls-certs\") pod \"kube-auth-proxy-68fcc45f6c-twgp9\" (UID: \"7eea523a-6a90-4f6b-9b89-c6c39bf6513f\") " pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-twgp9" Oct 30 11:58:51.638590 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.638562 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49hk2\" (UniqueName: \"kubernetes.io/projected/7eea523a-6a90-4f6b-9b89-c6c39bf6513f-kube-api-access-49hk2\") pod \"kube-auth-proxy-68fcc45f6c-twgp9\" (UID: \"7eea523a-6a90-4f6b-9b89-c6c39bf6513f\") " pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-twgp9" Oct 30 11:58:51.651464 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.651438 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-twgp9" Oct 30 11:58:51.806612 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.806571 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:51.952851 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:51.952809 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68fcc45f6c-twgp9"] Oct 30 11:58:51.955301 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:58:51.955241 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eea523a_6a90_4f6b_9b89_c6c39bf6513f.slice/crio-658fd5ca0c80467befd7c4b6c1bbbb3de4059db64fdecd77bd0bf12f321c87a2 WatchSource:0}: Error finding container 658fd5ca0c80467befd7c4b6c1bbbb3de4059db64fdecd77bd0bf12f321c87a2: Status 404 returned error can't find the container with id 658fd5ca0c80467befd7c4b6c1bbbb3de4059db64fdecd77bd0bf12f321c87a2 Oct 30 11:58:52.040149 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:52.040117 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:52.172915 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:52.172881 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:52.206455 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:52.206381 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-twgp9" event={"ID":"7eea523a-6a90-4f6b-9b89-c6c39bf6513f","Type":"ContainerStarted","Data":"658fd5ca0c80467befd7c4b6c1bbbb3de4059db64fdecd77bd0bf12f321c87a2"} Oct 30 11:58:53.138079 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:53.138037 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:54.036978 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:54.036943 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:54.163635 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:54.163600 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:55.224286 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:55.224231 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-twgp9" event={"ID":"7eea523a-6a90-4f6b-9b89-c6c39bf6513f","Type":"ContainerStarted","Data":"449ca419d041b04e19404124dc8caf65b660a73999f9f08d68d839311f46c2e6"} Oct 30 11:58:55.264390 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:55.264329 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68fcc45f6c-twgp9" podStartSLOduration=1.666562355 podStartE2EDuration="4.264315143s" totalImagesPullingTime="2.597752788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:55.264074281 +0000 UTC m=+288.373917175" watchObservedRunningTime="2026-10-30 11:58:55.264315143 +0000 UTC m=+288.374158058" Oct 30 11:58:56.230586 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:56.230500 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zrh8j" event={"ID":"6ec974e2-3c96-493d-b06a-f59ad7d60dce","Type":"ContainerStarted","Data":"f07f35ff5f7fdadff62ea6a97b8fc0f2ba6edffea63f8830829b1d35d7a0fe06"} Oct 30 11:58:56.268725 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:56.268679 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zrh8j" podStartSLOduration=1.380713779 podStartE2EDuration="6.268665561s" totalImagesPullingTime="4.887951782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:58:56.268546633 +0000 UTC m=+289.378389515" watchObservedRunningTime="2026-10-30 11:58:56.268665561 +0000 UTC m=+289.378508443" Oct 30 11:58:56.877403 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:56.877366 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:56.995989 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:56.995954 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:58.877832 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:58.877797 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:58:58.992436 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:58:58.992403 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:03.952609 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:03.952573 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:04.090332 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:04.090303 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:05.813753 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:59:05.813717 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 11:59:06.873211 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:06.873173 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:06.985973 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:06.985932 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:10.322188 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:10.322148 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:10.432912 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:10.432876 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:12.883639 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:12.883610 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:12.999593 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:12.999561 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:13.508974 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.508944 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f"] Oct 30 11:59:13.512477 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.512451 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" Oct 30 11:59:13.530169 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.530128 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 11:59:13.532166 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.532148 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f"] Oct 30 11:59:13.542141 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.542122 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-tbv9h\"" Oct 30 11:59:13.542383 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.542366 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 11:59:13.542383 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.542369 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 11:59:13.695589 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.695556 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fd3b7abd-89a3-4974-a1a6-2304f36cd082-manager-config\") pod \"jobset-controller-manager-f445d948b-brn6f\" (UID: \"fd3b7abd-89a3-4974-a1a6-2304f36cd082\") " pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" Oct 30 11:59:13.695767 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.695652 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd3b7abd-89a3-4974-a1a6-2304f36cd082-metrics-certs\") pod \"jobset-controller-manager-f445d948b-brn6f\" (UID: \"fd3b7abd-89a3-4974-a1a6-2304f36cd082\") " pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" Oct 30 11:59:13.695830 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.695781 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lnrb\" (UniqueName: \"kubernetes.io/projected/fd3b7abd-89a3-4974-a1a6-2304f36cd082-kube-api-access-7lnrb\") pod \"jobset-controller-manager-f445d948b-brn6f\" (UID: \"fd3b7abd-89a3-4974-a1a6-2304f36cd082\") " pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" Oct 30 11:59:13.695888 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.695839 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd3b7abd-89a3-4974-a1a6-2304f36cd082-cert\") pod \"jobset-controller-manager-f445d948b-brn6f\" (UID: \"fd3b7abd-89a3-4974-a1a6-2304f36cd082\") " pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" Oct 30 11:59:13.796649 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.796563 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd3b7abd-89a3-4974-a1a6-2304f36cd082-metrics-certs\") pod \"jobset-controller-manager-f445d948b-brn6f\" (UID: \"fd3b7abd-89a3-4974-a1a6-2304f36cd082\") " pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" Oct 30 11:59:13.796816 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.796657 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lnrb\" (UniqueName: \"kubernetes.io/projected/fd3b7abd-89a3-4974-a1a6-2304f36cd082-kube-api-access-7lnrb\") pod \"jobset-controller-manager-f445d948b-brn6f\" (UID: \"fd3b7abd-89a3-4974-a1a6-2304f36cd082\") " pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" Oct 30 11:59:13.796816 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.796692 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd3b7abd-89a3-4974-a1a6-2304f36cd082-cert\") pod \"jobset-controller-manager-f445d948b-brn6f\" (UID: \"fd3b7abd-89a3-4974-a1a6-2304f36cd082\") " pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" Oct 30 11:59:13.796816 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.796741 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fd3b7abd-89a3-4974-a1a6-2304f36cd082-manager-config\") pod \"jobset-controller-manager-f445d948b-brn6f\" (UID: \"fd3b7abd-89a3-4974-a1a6-2304f36cd082\") " pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" Oct 30 11:59:13.797630 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.797600 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fd3b7abd-89a3-4974-a1a6-2304f36cd082-manager-config\") pod \"jobset-controller-manager-f445d948b-brn6f\" (UID: \"fd3b7abd-89a3-4974-a1a6-2304f36cd082\") " pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" Oct 30 11:59:13.799707 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.799690 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd3b7abd-89a3-4974-a1a6-2304f36cd082-metrics-certs\") pod \"jobset-controller-manager-f445d948b-brn6f\" (UID: \"fd3b7abd-89a3-4974-a1a6-2304f36cd082\") " pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" Oct 30 11:59:13.799813 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.799729 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd3b7abd-89a3-4974-a1a6-2304f36cd082-cert\") pod \"jobset-controller-manager-f445d948b-brn6f\" (UID: \"fd3b7abd-89a3-4974-a1a6-2304f36cd082\") " pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" Oct 30 11:59:13.825129 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:13.825106 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lnrb\" (UniqueName: \"kubernetes.io/projected/fd3b7abd-89a3-4974-a1a6-2304f36cd082-kube-api-access-7lnrb\") pod \"jobset-controller-manager-f445d948b-brn6f\" (UID: \"fd3b7abd-89a3-4974-a1a6-2304f36cd082\") " pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" Oct 30 11:59:14.123321 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:14.123288 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" Oct 30 11:59:14.424573 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:14.424546 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f"] Oct 30 11:59:14.428011 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:59:14.427979 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3b7abd_89a3_4974_a1a6_2304f36cd082.slice/crio-8017609bad3cdff589fc5ced99c46e9ad117313d469bda36dcab2202d6568642 WatchSource:0}: Error finding container 8017609bad3cdff589fc5ced99c46e9ad117313d469bda36dcab2202d6568642: Status 404 returned error can't find the container with id 8017609bad3cdff589fc5ced99c46e9ad117313d469bda36dcab2202d6568642 Oct 30 11:59:14.430291 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:14.430242 4413 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 11:59:15.308305 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:15.308238 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" event={"ID":"fd3b7abd-89a3-4974-a1a6-2304f36cd082","Type":"ContainerStarted","Data":"8017609bad3cdff589fc5ced99c46e9ad117313d469bda36dcab2202d6568642"} Oct 30 11:59:16.149048 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:16.149010 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:16.275572 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:16.275540 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:16.398976 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:16.398918 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:17.319091 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:17.319054 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" event={"ID":"fd3b7abd-89a3-4974-a1a6-2304f36cd082","Type":"ContainerStarted","Data":"f96dd2e7220d890608f35d537ab411accc6b958ae5c4b4bf48d3ce331921f7c5"} Oct 30 11:59:17.319298 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:17.319186 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" Oct 30 11:59:17.360766 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:17.360708 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" podStartSLOduration=1.967522897 podStartE2EDuration="4.360692984s" totalImagesPullingTime="2.393170087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:17.359968672 +0000 UTC m=+310.469811553" watchObservedRunningTime="2026-10-30 11:59:17.360692984 +0000 UTC m=+310.470535866" Oct 30 11:59:23.164498 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:23.164386 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:23.359659 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:23.359624 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:28.328338 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:28.328302 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-f445d948b-brn6f" Oct 30 11:59:35.812476 ip-10-0-138-23 kubenswrapper[4413]: E1030 11:59:35.812439 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 11:59:39.479892 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:39.479855 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:39.599297 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:39.599241 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:54.164027 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:54.163990 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:55.069734 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.069691 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:55.189585 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.189557 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:55.310259 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.310216 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:55.444929 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.444879 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:55.456075 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.456043 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-g5g58"] Oct 30 11:59:55.463051 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.463033 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-g5g58" Oct 30 11:59:55.481517 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.481493 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-g5g58"] Oct 30 11:59:55.491478 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.491455 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 11:59:55.491609 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.491482 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 11:59:55.514243 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.514215 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-mhfr6"] Oct 30 11:59:55.517630 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.517617 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-mhfr6" Oct 30 11:59:55.536159 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.536140 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 11:59:55.536159 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.536145 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-g5vw7\"" Oct 30 11:59:55.557346 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.557322 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-mhfr6"] Oct 30 11:59:55.575454 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.575431 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a63b5bb9-3521-4f1a-bfdd-08325682cfb2-tls-certs\") pod \"model-serving-api-75bd89fdfc-g5g58\" (UID: \"a63b5bb9-3521-4f1a-bfdd-08325682cfb2\") " pod="opendatahub/model-serving-api-75bd89fdfc-g5g58" Oct 30 11:59:55.575565 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.575499 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf4v6\" (UniqueName: \"kubernetes.io/projected/a63b5bb9-3521-4f1a-bfdd-08325682cfb2-kube-api-access-cf4v6\") pod \"model-serving-api-75bd89fdfc-g5g58\" (UID: \"a63b5bb9-3521-4f1a-bfdd-08325682cfb2\") " pod="opendatahub/model-serving-api-75bd89fdfc-g5g58" Oct 30 11:59:55.627338 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.627313 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 11:59:55.627454 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.627419 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 11:59:55.627454 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.627446 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-d4rwn\"" Oct 30 11:59:55.651876 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.651844 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:55.676066 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.676044 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf4v6\" (UniqueName: \"kubernetes.io/projected/a63b5bb9-3521-4f1a-bfdd-08325682cfb2-kube-api-access-cf4v6\") pod \"model-serving-api-75bd89fdfc-g5g58\" (UID: \"a63b5bb9-3521-4f1a-bfdd-08325682cfb2\") " pod="opendatahub/model-serving-api-75bd89fdfc-g5g58" Oct 30 11:59:55.676170 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.676088 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e9919e6-4abf-45ba-bcee-0237e0577a06-cert\") pod \"odh-model-controller-769875db56-mhfr6\" (UID: \"7e9919e6-4abf-45ba-bcee-0237e0577a06\") " pod="opendatahub/odh-model-controller-769875db56-mhfr6" Oct 30 11:59:55.676170 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.676113 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2kqc\" (UniqueName: \"kubernetes.io/projected/7e9919e6-4abf-45ba-bcee-0237e0577a06-kube-api-access-p2kqc\") pod \"odh-model-controller-769875db56-mhfr6\" (UID: \"7e9919e6-4abf-45ba-bcee-0237e0577a06\") " pod="opendatahub/odh-model-controller-769875db56-mhfr6" Oct 30 11:59:55.676313 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.676198 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9919e6-4abf-45ba-bcee-0237e0577a06-metrics-certs\") pod \"odh-model-controller-769875db56-mhfr6\" (UID: \"7e9919e6-4abf-45ba-bcee-0237e0577a06\") " pod="opendatahub/odh-model-controller-769875db56-mhfr6" Oct 30 11:59:55.676313 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.676235 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a63b5bb9-3521-4f1a-bfdd-08325682cfb2-tls-certs\") pod \"model-serving-api-75bd89fdfc-g5g58\" (UID: \"a63b5bb9-3521-4f1a-bfdd-08325682cfb2\") " pod="opendatahub/model-serving-api-75bd89fdfc-g5g58" Oct 30 11:59:55.678768 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.678751 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a63b5bb9-3521-4f1a-bfdd-08325682cfb2-tls-certs\") pod \"model-serving-api-75bd89fdfc-g5g58\" (UID: \"a63b5bb9-3521-4f1a-bfdd-08325682cfb2\") " pod="opendatahub/model-serving-api-75bd89fdfc-g5g58" Oct 30 11:59:55.699143 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.699095 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf4v6\" (UniqueName: \"kubernetes.io/projected/a63b5bb9-3521-4f1a-bfdd-08325682cfb2-kube-api-access-cf4v6\") pod \"model-serving-api-75bd89fdfc-g5g58\" (UID: \"a63b5bb9-3521-4f1a-bfdd-08325682cfb2\") " pod="opendatahub/model-serving-api-75bd89fdfc-g5g58" Oct 30 11:59:55.775083 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.775042 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-g5g58" Oct 30 11:59:55.777418 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.777395 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2kqc\" (UniqueName: \"kubernetes.io/projected/7e9919e6-4abf-45ba-bcee-0237e0577a06-kube-api-access-p2kqc\") pod \"odh-model-controller-769875db56-mhfr6\" (UID: \"7e9919e6-4abf-45ba-bcee-0237e0577a06\") " pod="opendatahub/odh-model-controller-769875db56-mhfr6" Oct 30 11:59:55.777479 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.777437 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9919e6-4abf-45ba-bcee-0237e0577a06-metrics-certs\") pod \"odh-model-controller-769875db56-mhfr6\" (UID: \"7e9919e6-4abf-45ba-bcee-0237e0577a06\") " pod="opendatahub/odh-model-controller-769875db56-mhfr6" Oct 30 11:59:55.777521 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.777510 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e9919e6-4abf-45ba-bcee-0237e0577a06-cert\") pod \"odh-model-controller-769875db56-mhfr6\" (UID: \"7e9919e6-4abf-45ba-bcee-0237e0577a06\") " pod="opendatahub/odh-model-controller-769875db56-mhfr6" Oct 30 11:59:55.780083 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.780061 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9919e6-4abf-45ba-bcee-0237e0577a06-metrics-certs\") pod \"odh-model-controller-769875db56-mhfr6\" (UID: \"7e9919e6-4abf-45ba-bcee-0237e0577a06\") " pod="opendatahub/odh-model-controller-769875db56-mhfr6" Oct 30 11:59:55.780274 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.780232 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e9919e6-4abf-45ba-bcee-0237e0577a06-cert\") pod \"odh-model-controller-769875db56-mhfr6\" (UID: \"7e9919e6-4abf-45ba-bcee-0237e0577a06\") " pod="opendatahub/odh-model-controller-769875db56-mhfr6" Oct 30 11:59:55.801916 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.801883 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2kqc\" (UniqueName: \"kubernetes.io/projected/7e9919e6-4abf-45ba-bcee-0237e0577a06-kube-api-access-p2kqc\") pod \"odh-model-controller-769875db56-mhfr6\" (UID: \"7e9919e6-4abf-45ba-bcee-0237e0577a06\") " pod="opendatahub/odh-model-controller-769875db56-mhfr6" Oct 30 11:59:55.831948 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.829529 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-mhfr6" Oct 30 11:59:55.936123 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:55.936096 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:56.076702 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:56.076677 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-g5g58"] Oct 30 11:59:56.079013 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:59:56.078985 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63b5bb9_3521_4f1a_bfdd_08325682cfb2.slice/crio-77b6afd82cdca60f116626cb3b3d8b916c0c3e44e778b8b9d39e2e41c3571593 WatchSource:0}: Error finding container 77b6afd82cdca60f116626cb3b3d8b916c0c3e44e778b8b9d39e2e41c3571593: Status 404 returned error can't find the container with id 77b6afd82cdca60f116626cb3b3d8b916c0c3e44e778b8b9d39e2e41c3571593 Oct 30 11:59:56.130777 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:56.130743 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-mhfr6"] Oct 30 11:59:56.132783 ip-10-0-138-23 kubenswrapper[4413]: W1030 11:59:56.132760 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9919e6_4abf_45ba_bcee_0237e0577a06.slice/crio-547413ff0895ab2b33e547d0cb30abce42ad990e4bb24418a65d9d2cee7638dd WatchSource:0}: Error finding container 547413ff0895ab2b33e547d0cb30abce42ad990e4bb24418a65d9d2cee7638dd: Status 404 returned error can't find the container with id 547413ff0895ab2b33e547d0cb30abce42ad990e4bb24418a65d9d2cee7638dd Oct 30 11:59:56.490707 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:56.490660 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-mhfr6" event={"ID":"7e9919e6-4abf-45ba-bcee-0237e0577a06","Type":"ContainerStarted","Data":"547413ff0895ab2b33e547d0cb30abce42ad990e4bb24418a65d9d2cee7638dd"} Oct 30 11:59:56.491854 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:56.491822 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-g5g58" event={"ID":"a63b5bb9-3521-4f1a-bfdd-08325682cfb2","Type":"ContainerStarted","Data":"77b6afd82cdca60f116626cb3b3d8b916c0c3e44e778b8b9d39e2e41c3571593"} Oct 30 11:59:57.935294 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:57.935237 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:58.069296 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:58.069240 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:58.200544 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:58.200467 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:58.345517 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:58.345488 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:58.477174 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:58.477079 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 11:59:58.505224 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:58.505183 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-g5g58" event={"ID":"a63b5bb9-3521-4f1a-bfdd-08325682cfb2","Type":"ContainerStarted","Data":"d73f08642d5c701b19e60e1c5b1b4eb1056d2b329bc45f21893d96a6c7c7c613"} Oct 30 11:59:58.505428 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:58.505412 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-g5g58" Oct 30 11:59:58.547609 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:58.547552 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-g5g58" podStartSLOduration=1.325735447 podStartE2EDuration="3.547534961s" totalImagesPullingTime="2.221799514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:58.546008771 +0000 UTC m=+351.655851653" watchObservedRunningTime="2026-10-30 11:59:58.547534961 +0000 UTC m=+351.657377842" Oct 30 11:59:59.511148 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:59.511111 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-mhfr6" event={"ID":"7e9919e6-4abf-45ba-bcee-0237e0577a06","Type":"ContainerStarted","Data":"ae715d16f755cf2f525bae4f27ed5c9b7423c53c0beb1ed21b6884b4fd732517"} Oct 30 11:59:59.511602 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:59.511375 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-mhfr6" Oct 30 11:59:59.555121 ip-10-0-138-23 kubenswrapper[4413]: I1030 11:59:59.555070 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-mhfr6" podStartSLOduration=1.8459170440000001 podStartE2EDuration="4.555056067s" totalImagesPullingTime="2.709139023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:59.552301807 +0000 UTC m=+352.662144688" watchObservedRunningTime="2026-10-30 11:59:59.555056067 +0000 UTC m=+352.664898948" Oct 30 12:00:05.814099 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:00:05.814055 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:00:09.515776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:00:09.515742 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-g5g58" Oct 30 12:00:10.518402 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:00:10.518374 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-mhfr6" Oct 30 12:00:35.814026 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:00:35.813988 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:00:54.265667 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:00:54.265578 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:01:05.813136 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:01:05.813090 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:01:09.695281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:01:09.695212 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:01:09.814004 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:01:09.813972 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:01:12.543557 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:01:12.543512 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:01:35.814000 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:01:35.813955 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:01:54.434305 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:01:54.434274 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:01:57.334729 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:01:57.334697 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:01:57.452052 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:01:57.452017 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:02:05.813901 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:02:05.813864 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:02:35.813340 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:02:35.813296 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:02:54.532795 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:02:54.532764 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:03:05.812835 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:03:05.812799 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:03:06.731375 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:03:06.731311 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:03:06.854381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:03:06.854319 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:03:35.814077 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:03:35.814036 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:03:44.798851 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:03:44.798817 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:03:54.593357 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:03:54.593263 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:04:05.812656 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:04:05.812611 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:04:35.813593 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:04:35.813550 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:04:51.072058 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:04:51.072019 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:04:54.737703 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:04:54.737655 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:05:05.814965 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:05:05.814918 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:05:35.814133 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:05:35.814093 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:05:54.883616 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:05:54.883582 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:06:05.812850 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:06:05.812807 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:06:12.757735 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:06:12.757705 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:06:35.812031 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:06:35.811991 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:06:54.995211 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:06:54.995128 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:07:05.815005 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:07:05.814961 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:07:35.813459 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:07:35.813425 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:07:55.139979 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:07:55.139947 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:07:55.319195 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:07:55.319164 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:08:05.813172 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:08:05.813139 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:08:06.222567 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:08:06.222521 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:08:31.968541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:08:31.968478 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:08:35.813622 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:08:35.813588 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:08:55.220351 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:08:55.220311 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:09:05.813393 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:09:05.813353 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:09:30.988267 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:09:30.988223 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:09:31.196992 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:09:31.196963 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:09:31.316904 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:09:31.316813 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:09:31.950654 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:09:31.950622 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:09:32.065512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:09:32.065474 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:09:35.812283 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:09:35.812234 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:09:51.672518 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:09:51.672448 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:09:51.811114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:09:51.811069 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:09:52.386408 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:09:52.386365 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:09:55.360352 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:09:55.360319 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:10:05.814758 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:10:05.814719 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:10:12.147661 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:12.147626 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:10:12.273294 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:12.273242 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:10:12.392161 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:12.392126 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:10:12.516424 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:12.516326 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:10:15.755498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:15.755459 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:10:15.877377 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:15.877332 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:10:17.767501 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:17.767465 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:10:17.878041 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:17.878009 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:10:35.814312 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:10:35.814271 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:10:48.206628 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:48.206597 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:10:48.328126 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:48.328097 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:10:49.846623 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:49.846587 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:10:49.963286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:49.963233 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:10:55.466836 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:55.466781 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:10:59.381586 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:59.381555 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:10:59.510394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:59.510344 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:10:59.908711 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:59.908679 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj"] Oct 30 12:10:59.912084 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:59.912069 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" Oct 30 12:10:59.929404 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:59.929379 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 12:10:59.942942 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:59.942923 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj"] Oct 30 12:10:59.986034 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:59.986012 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bed695ee-4b0b-42cc-9c1f-f658517d9b09-maas-api-tls\") pod \"maas-api-57b6f6b66d-kcfsj\" (UID: \"bed695ee-4b0b-42cc-9c1f-f658517d9b09\") " pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" Oct 30 12:10:59.986142 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:59.986059 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgbll\" (UniqueName: \"kubernetes.io/projected/bed695ee-4b0b-42cc-9c1f-f658517d9b09-kube-api-access-vgbll\") pod \"maas-api-57b6f6b66d-kcfsj\" (UID: \"bed695ee-4b0b-42cc-9c1f-f658517d9b09\") " pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" Oct 30 12:10:59.986142 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:59.986088 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bed695ee-4b0b-42cc-9c1f-f658517d9b09-openshift-service-ca\") pod \"maas-api-57b6f6b66d-kcfsj\" (UID: \"bed695ee-4b0b-42cc-9c1f-f658517d9b09\") " pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" Oct 30 12:10:59.986142 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:59.986120 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bed695ee-4b0b-42cc-9c1f-f658517d9b09-metrics-certs\") pod \"maas-api-57b6f6b66d-kcfsj\" (UID: \"bed695ee-4b0b-42cc-9c1f-f658517d9b09\") " pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" Oct 30 12:10:59.996491 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:59.996462 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 12:10:59.996491 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:10:59.996490 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 12:11:00.027729 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:00.027709 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rtvpm\"" Oct 30 12:11:00.027876 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:00.027858 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 12:11:00.087130 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:00.087110 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgbll\" (UniqueName: \"kubernetes.io/projected/bed695ee-4b0b-42cc-9c1f-f658517d9b09-kube-api-access-vgbll\") pod \"maas-api-57b6f6b66d-kcfsj\" (UID: \"bed695ee-4b0b-42cc-9c1f-f658517d9b09\") " pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" Oct 30 12:11:00.087290 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:00.087141 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bed695ee-4b0b-42cc-9c1f-f658517d9b09-openshift-service-ca\") pod \"maas-api-57b6f6b66d-kcfsj\" (UID: \"bed695ee-4b0b-42cc-9c1f-f658517d9b09\") " pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" Oct 30 12:11:00.087290 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:00.087159 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bed695ee-4b0b-42cc-9c1f-f658517d9b09-metrics-certs\") pod \"maas-api-57b6f6b66d-kcfsj\" (UID: \"bed695ee-4b0b-42cc-9c1f-f658517d9b09\") " pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" Oct 30 12:11:00.087290 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:00.087203 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bed695ee-4b0b-42cc-9c1f-f658517d9b09-maas-api-tls\") pod \"maas-api-57b6f6b66d-kcfsj\" (UID: \"bed695ee-4b0b-42cc-9c1f-f658517d9b09\") " pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" Oct 30 12:11:00.087882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:00.087862 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bed695ee-4b0b-42cc-9c1f-f658517d9b09-openshift-service-ca\") pod \"maas-api-57b6f6b66d-kcfsj\" (UID: \"bed695ee-4b0b-42cc-9c1f-f658517d9b09\") " pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" Oct 30 12:11:00.089990 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:00.089969 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bed695ee-4b0b-42cc-9c1f-f658517d9b09-metrics-certs\") pod \"maas-api-57b6f6b66d-kcfsj\" (UID: \"bed695ee-4b0b-42cc-9c1f-f658517d9b09\") " pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" Oct 30 12:11:00.090090 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:00.090008 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bed695ee-4b0b-42cc-9c1f-f658517d9b09-maas-api-tls\") pod \"maas-api-57b6f6b66d-kcfsj\" (UID: \"bed695ee-4b0b-42cc-9c1f-f658517d9b09\") " pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" Oct 30 12:11:00.111954 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:00.111922 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgbll\" (UniqueName: \"kubernetes.io/projected/bed695ee-4b0b-42cc-9c1f-f658517d9b09-kube-api-access-vgbll\") pod \"maas-api-57b6f6b66d-kcfsj\" (UID: \"bed695ee-4b0b-42cc-9c1f-f658517d9b09\") " pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" Oct 30 12:11:00.224922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:00.224829 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" Oct 30 12:11:00.522654 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:00.522633 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj"] Oct 30 12:11:00.524982 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:11:00.524956 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed695ee_4b0b_42cc_9c1f_f658517d9b09.slice/crio-66d79127d8230c77c1a3f402eaec5d42fe8ebfe12c295205097dbff85bf5af60 WatchSource:0}: Error finding container 66d79127d8230c77c1a3f402eaec5d42fe8ebfe12c295205097dbff85bf5af60: Status 404 returned error can't find the container with id 66d79127d8230c77c1a3f402eaec5d42fe8ebfe12c295205097dbff85bf5af60 Oct 30 12:11:00.526228 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:00.526211 4413 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:11:01.365690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:01.365655 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" event={"ID":"bed695ee-4b0b-42cc-9c1f-f658517d9b09","Type":"ContainerStarted","Data":"66d79127d8230c77c1a3f402eaec5d42fe8ebfe12c295205097dbff85bf5af60"} Oct 30 12:11:03.376083 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:03.376050 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" event={"ID":"bed695ee-4b0b-42cc-9c1f-f658517d9b09","Type":"ContainerStarted","Data":"3520c94e0206afacb25e3a962c3e04aa0a138fb7d99e53c8e7e41fa0ea47f57b"} Oct 30 12:11:03.376472 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:03.376175 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" Oct 30 12:11:03.420439 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:03.420393 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" podStartSLOduration=2.110250748 podStartE2EDuration="4.420379558s" totalImagesPullingTime="2.31012881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:03.418667357 +0000 UTC m=+1016.528510239" watchObservedRunningTime="2026-10-30 12:11:03.420379558 +0000 UTC m=+1016.530222447" Oct 30 12:11:05.812678 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:11:05.812634 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:11:09.385355 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:09.385328 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-57b6f6b66d-kcfsj" Oct 30 12:11:25.155163 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.155125 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6"] Oct 30 12:11:25.158907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.158885 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.174343 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.174322 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 12:11:25.188236 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.188203 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6"] Oct 30 12:11:25.265940 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.265909 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lkcrv\"" Oct 30 12:11:25.314193 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.314162 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.314418 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.314203 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.314418 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.314223 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.314418 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.314243 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6rs6\" (UniqueName: \"kubernetes.io/projected/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-kube-api-access-j6rs6\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.314418 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.314383 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.314418 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.314418 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.314621 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.314445 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.314621 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.314510 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.314621 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.314557 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.415819 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.415733 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.415819 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.415772 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.415819 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.415801 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6rs6\" (UniqueName: \"kubernetes.io/projected/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-kube-api-access-j6rs6\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.416108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.415850 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.416108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.415896 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.416108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.415932 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.416108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.415963 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.416108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.416018 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.416108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.416055 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.416460 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.416157 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.416460 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.416307 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.416614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.416486 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.416614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.416548 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.416797 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.416779 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.418932 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.418911 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.419133 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.419116 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.440651 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.440629 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.445708 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.445686 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6rs6\" (UniqueName: \"kubernetes.io/projected/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-kube-api-access-j6rs6\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-b94h6\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.471859 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.471840 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:25.786760 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:25.786735 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6"] Oct 30 12:11:25.788836 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:11:25.788813 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda179c4e6_0f4b_4ebb_9d08_f253b8d26e58.slice/crio-f1675eecfb1225e51049dc65ed1480aa6f1644b0d0d41d317b8ddbb4cdc3099a WatchSource:0}: Error finding container f1675eecfb1225e51049dc65ed1480aa6f1644b0d0d41d317b8ddbb4cdc3099a: Status 404 returned error can't find the container with id f1675eecfb1225e51049dc65ed1480aa6f1644b0d0d41d317b8ddbb4cdc3099a Oct 30 12:11:26.482379 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:26.482346 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" event={"ID":"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58","Type":"ContainerStarted","Data":"f1675eecfb1225e51049dc65ed1480aa6f1644b0d0d41d317b8ddbb4cdc3099a"} Oct 30 12:11:28.376997 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:28.376966 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:11:28.377335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:28.377034 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:11:28.377335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:28.377062 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:11:28.495023 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:28.494985 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" event={"ID":"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58","Type":"ContainerStarted","Data":"24658298814e25db7b926dcd5ce51d6174f544ba63d05d0e72468c54183aec8f"} Oct 30 12:11:28.564030 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:28.563985 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" podStartSLOduration=0.97853414 podStartE2EDuration="3.563970794s" totalImagesPullingTime="2.585436654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:28.551435477 +0000 UTC m=+1041.661278371" watchObservedRunningTime="2026-10-30 12:11:28.563970794 +0000 UTC m=+1041.673813674" Oct 30 12:11:29.472415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:29.472382 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:29.473736 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:29.473712 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-b94h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 12:11:29.473803 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:29.473763 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" podUID="a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 12:11:30.472768 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:30.472731 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-b94h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 12:11:30.473154 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:30.472780 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" podUID="a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 12:11:31.472239 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:31.472204 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-b94h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 12:11:31.472429 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:31.472280 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" podUID="a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 12:11:32.473104 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:32.473070 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-b94h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 12:11:32.473573 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:32.473125 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" podUID="a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 12:11:33.472816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:33.472777 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-b94h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 12:11:33.473043 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:33.472845 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" podUID="a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 12:11:34.476468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:34.476436 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:34.476972 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:34.476813 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:34.477469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:34.477450 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:11:35.351851 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:35.351819 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:11:35.812206 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:11:35.812170 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:11:37.272665 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:37.272630 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:11:37.391526 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:37.391491 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:11:40.174511 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.174477 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8"] Oct 30 12:11:40.180992 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.180974 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.195215 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.195197 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps"] Oct 30 12:11:40.198682 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.198665 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.207968 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.207945 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8"] Oct 30 12:11:40.231950 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.231924 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps"] Oct 30 12:11:40.271060 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.271016 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d21fd6-1a62-4b69-af88-d5038a533f90-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.271221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.271080 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d21fd6-1a62-4b69-af88-d5038a533f90-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.271221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.271114 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.271221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.271141 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d21fd6-1a62-4b69-af88-d5038a533f90-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.271221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.271176 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d21fd6-1a62-4b69-af88-d5038a533f90-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.271221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.271219 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbpvh\" (UniqueName: \"kubernetes.io/projected/6662310d-7d7f-424e-a1c8-45988ed0db96-kube-api-access-xbpvh\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.271492 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.271274 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.271492 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.271312 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.271492 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.271346 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d21fd6-1a62-4b69-af88-d5038a533f90-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.271492 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.271396 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.271492 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.271431 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6662310d-7d7f-424e-a1c8-45988ed0db96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.271492 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.271462 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpbq2\" (UniqueName: \"kubernetes.io/projected/b2d21fd6-1a62-4b69-af88-d5038a533f90-kube-api-access-cpbq2\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.271742 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.271515 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d21fd6-1a62-4b69-af88-d5038a533f90-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.271742 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.271551 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.271742 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.271602 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d21fd6-1a62-4b69-af88-d5038a533f90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.271742 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.271637 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d21fd6-1a62-4b69-af88-d5038a533f90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.271742 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.271692 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.271742 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.271724 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.372831 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.372795 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d21fd6-1a62-4b69-af88-d5038a533f90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.373004 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.372850 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.373004 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.372884 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.373004 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.372908 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d21fd6-1a62-4b69-af88-d5038a533f90-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.373160 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373082 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d21fd6-1a62-4b69-af88-d5038a533f90-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.373160 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373130 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.373309 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373160 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d21fd6-1a62-4b69-af88-d5038a533f90-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.373309 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373188 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d21fd6-1a62-4b69-af88-d5038a533f90-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.373309 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373234 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbpvh\" (UniqueName: \"kubernetes.io/projected/6662310d-7d7f-424e-a1c8-45988ed0db96-kube-api-access-xbpvh\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.373474 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373363 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.373474 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373407 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d21fd6-1a62-4b69-af88-d5038a533f90-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.373474 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373463 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.373629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373513 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.373629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373513 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d21fd6-1a62-4b69-af88-d5038a533f90-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.373629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373562 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d21fd6-1a62-4b69-af88-d5038a533f90-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.373629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373606 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.373629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373608 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.373629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373594 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d21fd6-1a62-4b69-af88-d5038a533f90-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.373947 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373636 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6662310d-7d7f-424e-a1c8-45988ed0db96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.373947 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373675 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpbq2\" (UniqueName: \"kubernetes.io/projected/b2d21fd6-1a62-4b69-af88-d5038a533f90-kube-api-access-cpbq2\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.373947 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373724 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d21fd6-1a62-4b69-af88-d5038a533f90-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.373947 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373760 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.373947 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373782 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.373947 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373808 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d21fd6-1a62-4b69-af88-d5038a533f90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.373947 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.373811 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d21fd6-1a62-4b69-af88-d5038a533f90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.374478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.374025 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d21fd6-1a62-4b69-af88-d5038a533f90-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.374478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.374365 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.374590 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.374495 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6662310d-7d7f-424e-a1c8-45988ed0db96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.375908 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.375882 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.376011 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.375913 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d21fd6-1a62-4b69-af88-d5038a533f90-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.376204 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.376183 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.376388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.376374 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d21fd6-1a62-4b69-af88-d5038a533f90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.396362 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.396335 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d21fd6-1a62-4b69-af88-d5038a533f90-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.396538 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.396517 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.396622 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.396606 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbpvh\" (UniqueName: \"kubernetes.io/projected/6662310d-7d7f-424e-a1c8-45988ed0db96-kube-api-access-xbpvh\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w9qb8\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.396681 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.396639 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpbq2\" (UniqueName: \"kubernetes.io/projected/b2d21fd6-1a62-4b69-af88-d5038a533f90-kube-api-access-cpbq2\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9x4ps\" (UID: \"b2d21fd6-1a62-4b69-af88-d5038a533f90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.494802 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.494710 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:40.512846 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.512819 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:40.809957 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.809881 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8"] Oct 30 12:11:40.813153 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:11:40.813127 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6662310d_7d7f_424e_a1c8_45988ed0db96.slice/crio-968e98f4cf33a5a736fdca643586162e8373b696bdbcb6ec9378bd843bb6df0b WatchSource:0}: Error finding container 968e98f4cf33a5a736fdca643586162e8373b696bdbcb6ec9378bd843bb6df0b: Status 404 returned error can't find the container with id 968e98f4cf33a5a736fdca643586162e8373b696bdbcb6ec9378bd843bb6df0b Oct 30 12:11:40.815379 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.815346 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:11:40.815445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.815409 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:11:40.815445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.815437 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:11:40.839710 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.839690 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps"] Oct 30 12:11:40.843342 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:11:40.843312 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d21fd6_1a62_4b69_af88_d5038a533f90.slice/crio-acce4752c266639b1a207a3740be1c113a7ec495722817a4c93d0f76f564ef28 WatchSource:0}: Error finding container acce4752c266639b1a207a3740be1c113a7ec495722817a4c93d0f76f564ef28: Status 404 returned error can't find the container with id acce4752c266639b1a207a3740be1c113a7ec495722817a4c93d0f76f564ef28 Oct 30 12:11:40.845532 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.845498 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:11:40.845611 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.845567 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:11:40.845611 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:40.845597 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:11:41.463929 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:41.463898 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:11:41.573892 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:41.573854 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" event={"ID":"b2d21fd6-1a62-4b69-af88-d5038a533f90","Type":"ContainerStarted","Data":"8fc62cd526e96dd2be4650bb078c291a70d07cd7826572057e109215d3d07ea5"} Oct 30 12:11:41.574093 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:41.573926 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" event={"ID":"b2d21fd6-1a62-4b69-af88-d5038a533f90","Type":"ContainerStarted","Data":"acce4752c266639b1a207a3740be1c113a7ec495722817a4c93d0f76f564ef28"} Oct 30 12:11:41.575722 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:41.575693 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" event={"ID":"6662310d-7d7f-424e-a1c8-45988ed0db96","Type":"ContainerStarted","Data":"875f3ceae94422cf23a6b22b7f20badedf7e530818f607535c7c2f908f403c35"} Oct 30 12:11:41.575865 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:41.575745 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" event={"ID":"6662310d-7d7f-424e-a1c8-45988ed0db96","Type":"ContainerStarted","Data":"968e98f4cf33a5a736fdca643586162e8373b696bdbcb6ec9378bd843bb6df0b"} Oct 30 12:11:41.587198 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:41.587175 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:11:41.648608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:41.645940 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" podStartSLOduration=1.645916696 podStartE2EDuration="1.645916696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:41.619816677 +0000 UTC m=+1054.729659561" watchObservedRunningTime="2026-10-30 12:11:41.645916696 +0000 UTC m=+1054.755759580" Oct 30 12:11:41.677796 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:41.677745 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" podStartSLOduration=1.6777308359999998 podStartE2EDuration="1.677730836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:11:41.677043817 +0000 UTC m=+1054.786886695" watchObservedRunningTime="2026-10-30 12:11:41.677730836 +0000 UTC m=+1054.787573773" Oct 30 12:11:42.498628 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:42.498598 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:42.502232 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:42.501864 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 12:11:42.502232 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:42.501926 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" podUID="6662310d-7d7f-424e-a1c8-45988ed0db96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 12:11:42.515278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:42.514561 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:42.516608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:42.516494 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 12:11:42.516608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:42.516546 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" podUID="b2d21fd6-1a62-4b69-af88-d5038a533f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 12:11:43.513395 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:43.513349 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 12:11:43.513924 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:43.513421 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" podUID="6662310d-7d7f-424e-a1c8-45988ed0db96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 12:11:43.514242 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:43.514219 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 12:11:43.514436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:43.514413 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" podUID="b2d21fd6-1a62-4b69-af88-d5038a533f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 12:11:44.495700 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:44.495663 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 12:11:44.495979 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:44.495950 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" podUID="6662310d-7d7f-424e-a1c8-45988ed0db96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 12:11:44.517210 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:44.517177 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 12:11:44.517828 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:44.517798 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" podUID="b2d21fd6-1a62-4b69-af88-d5038a533f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 12:11:45.502282 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:45.498673 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 12:11:45.502282 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:45.498757 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" podUID="6662310d-7d7f-424e-a1c8-45988ed0db96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 12:11:45.515713 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:45.515677 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 12:11:45.515983 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:45.515950 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" podUID="b2d21fd6-1a62-4b69-af88-d5038a533f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 12:11:46.502278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:46.498654 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 12:11:46.502278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:46.498715 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" podUID="6662310d-7d7f-424e-a1c8-45988ed0db96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 12:11:46.514286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:46.513663 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 12:11:46.514286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:46.513742 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" podUID="b2d21fd6-1a62-4b69-af88-d5038a533f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 12:11:47.500625 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:47.500592 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:47.501007 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:47.500986 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:47.501669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:47.501655 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:11:47.517185 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:47.517157 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:47.517592 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:47.517575 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:47.518322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:47.518299 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" Oct 30 12:11:55.592600 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:11:55.592571 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:12:05.813816 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:12:05.813774 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:12:06.145715 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:06.145684 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:12:06.558877 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:06.558792 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:12:07.900687 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:07.900653 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:12:08.085074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:08.085048 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:12:13.901204 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:13.901174 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:12:19.355431 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:19.355397 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:12:21.666600 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.666566 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m"] Oct 30 12:12:21.675832 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.675802 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.704667 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.704635 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 12:12:21.704667 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.704646 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tqhfc\"" Oct 30 12:12:21.706112 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.706088 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m"] Oct 30 12:12:21.709001 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.708984 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 12:12:21.709068 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.709005 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 12:12:21.723614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.723590 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b95ec1b9-803b-428c-b6ab-eef315fd03e5-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wvt6m\" (UID: \"b95ec1b9-803b-428c-b6ab-eef315fd03e5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.723715 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.723640 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b95ec1b9-803b-428c-b6ab-eef315fd03e5-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wvt6m\" (UID: \"b95ec1b9-803b-428c-b6ab-eef315fd03e5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.723787 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.723728 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b95ec1b9-803b-428c-b6ab-eef315fd03e5-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wvt6m\" (UID: \"b95ec1b9-803b-428c-b6ab-eef315fd03e5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.723787 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.723766 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b95ec1b9-803b-428c-b6ab-eef315fd03e5-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wvt6m\" (UID: \"b95ec1b9-803b-428c-b6ab-eef315fd03e5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.723864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.723839 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b95ec1b9-803b-428c-b6ab-eef315fd03e5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wvt6m\" (UID: \"b95ec1b9-803b-428c-b6ab-eef315fd03e5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.723901 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.723883 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg6d2\" (UniqueName: \"kubernetes.io/projected/b95ec1b9-803b-428c-b6ab-eef315fd03e5-kube-api-access-sg6d2\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wvt6m\" (UID: \"b95ec1b9-803b-428c-b6ab-eef315fd03e5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.725863 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.725845 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:12:21.824315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.824287 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b95ec1b9-803b-428c-b6ab-eef315fd03e5-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wvt6m\" (UID: \"b95ec1b9-803b-428c-b6ab-eef315fd03e5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.824444 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.824334 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b95ec1b9-803b-428c-b6ab-eef315fd03e5-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wvt6m\" (UID: \"b95ec1b9-803b-428c-b6ab-eef315fd03e5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.824509 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.824458 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b95ec1b9-803b-428c-b6ab-eef315fd03e5-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wvt6m\" (UID: \"b95ec1b9-803b-428c-b6ab-eef315fd03e5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.824509 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.824490 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b95ec1b9-803b-428c-b6ab-eef315fd03e5-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wvt6m\" (UID: \"b95ec1b9-803b-428c-b6ab-eef315fd03e5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.824611 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.824527 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b95ec1b9-803b-428c-b6ab-eef315fd03e5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wvt6m\" (UID: \"b95ec1b9-803b-428c-b6ab-eef315fd03e5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.824611 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.824576 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg6d2\" (UniqueName: \"kubernetes.io/projected/b95ec1b9-803b-428c-b6ab-eef315fd03e5-kube-api-access-sg6d2\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wvt6m\" (UID: \"b95ec1b9-803b-428c-b6ab-eef315fd03e5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.824848 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.824823 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b95ec1b9-803b-428c-b6ab-eef315fd03e5-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wvt6m\" (UID: \"b95ec1b9-803b-428c-b6ab-eef315fd03e5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.824934 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.824863 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b95ec1b9-803b-428c-b6ab-eef315fd03e5-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wvt6m\" (UID: \"b95ec1b9-803b-428c-b6ab-eef315fd03e5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.824934 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.824885 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b95ec1b9-803b-428c-b6ab-eef315fd03e5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wvt6m\" (UID: \"b95ec1b9-803b-428c-b6ab-eef315fd03e5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.826896 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.826875 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b95ec1b9-803b-428c-b6ab-eef315fd03e5-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wvt6m\" (UID: \"b95ec1b9-803b-428c-b6ab-eef315fd03e5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.827185 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.827168 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b95ec1b9-803b-428c-b6ab-eef315fd03e5-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wvt6m\" (UID: \"b95ec1b9-803b-428c-b6ab-eef315fd03e5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.856093 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.856062 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg6d2\" (UniqueName: \"kubernetes.io/projected/b95ec1b9-803b-428c-b6ab-eef315fd03e5-kube-api-access-sg6d2\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wvt6m\" (UID: \"b95ec1b9-803b-428c-b6ab-eef315fd03e5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:21.989357 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:21.989280 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:22.287320 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:22.287294 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m"] Oct 30 12:12:22.289983 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:12:22.289956 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95ec1b9_803b_428c_b6ab_eef315fd03e5.slice/crio-bec4bac0061fc0906256b6ab000cf2f4760df51d44dbac27eb82514c7785b270 WatchSource:0}: Error finding container bec4bac0061fc0906256b6ab000cf2f4760df51d44dbac27eb82514c7785b270: Status 404 returned error can't find the container with id bec4bac0061fc0906256b6ab000cf2f4760df51d44dbac27eb82514c7785b270 Oct 30 12:12:22.793709 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:22.793628 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" event={"ID":"b95ec1b9-803b-428c-b6ab-eef315fd03e5","Type":"ContainerStarted","Data":"bec4bac0061fc0906256b6ab000cf2f4760df51d44dbac27eb82514c7785b270"} Oct 30 12:12:23.685846 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:23.685814 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:12:23.857436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:23.857387 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" event={"ID":"b95ec1b9-803b-428c-b6ab-eef315fd03e5","Type":"ContainerStarted","Data":"029b015eaf4ac5565d0302c0c38c35365016973b8e937eb7c571231aac2f091a"} Oct 30 12:12:23.941329 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:23.919227 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" podStartSLOduration=1.857318062 podStartE2EDuration="2.919206077s" totalImagesPullingTime="1.061888015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:23.915304571 +0000 UTC m=+1097.025147453" watchObservedRunningTime="2026-10-30 12:12:23.919206077 +0000 UTC m=+1097.029048958" Oct 30 12:12:24.010274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:24.009361 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:25.570095 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.570067 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg"] Oct 30 12:12:25.574304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.574282 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.601511 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.601484 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 12:12:25.603116 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.603092 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg"] Oct 30 12:12:25.633193 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.633162 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:12:25.670355 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.670328 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d56b1153-6a1c-45c5-bd51-6e8b40ef58ec-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg\" (UID: \"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.670458 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.670389 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d56b1153-6a1c-45c5-bd51-6e8b40ef58ec-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg\" (UID: \"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.670498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.670479 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d56b1153-6a1c-45c5-bd51-6e8b40ef58ec-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg\" (UID: \"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.670536 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.670507 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlj48\" (UniqueName: \"kubernetes.io/projected/d56b1153-6a1c-45c5-bd51-6e8b40ef58ec-kube-api-access-qlj48\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg\" (UID: \"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.670570 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.670532 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d56b1153-6a1c-45c5-bd51-6e8b40ef58ec-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg\" (UID: \"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.670602 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.670585 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d56b1153-6a1c-45c5-bd51-6e8b40ef58ec-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg\" (UID: \"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.771262 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.771204 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d56b1153-6a1c-45c5-bd51-6e8b40ef58ec-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg\" (UID: \"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.771451 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.771357 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d56b1153-6a1c-45c5-bd51-6e8b40ef58ec-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg\" (UID: \"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.771451 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.771417 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d56b1153-6a1c-45c5-bd51-6e8b40ef58ec-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg\" (UID: \"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.771565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.771468 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlj48\" (UniqueName: \"kubernetes.io/projected/d56b1153-6a1c-45c5-bd51-6e8b40ef58ec-kube-api-access-qlj48\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg\" (UID: \"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.771565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.771505 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d56b1153-6a1c-45c5-bd51-6e8b40ef58ec-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg\" (UID: \"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.771565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.771531 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d56b1153-6a1c-45c5-bd51-6e8b40ef58ec-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg\" (UID: \"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.771810 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.771786 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d56b1153-6a1c-45c5-bd51-6e8b40ef58ec-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg\" (UID: \"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.772381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.771890 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d56b1153-6a1c-45c5-bd51-6e8b40ef58ec-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg\" (UID: \"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.772381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.771971 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d56b1153-6a1c-45c5-bd51-6e8b40ef58ec-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg\" (UID: \"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.774416 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.774393 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d56b1153-6a1c-45c5-bd51-6e8b40ef58ec-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg\" (UID: \"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.782944 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.782912 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d56b1153-6a1c-45c5-bd51-6e8b40ef58ec-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg\" (UID: \"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.798648 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.798620 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlj48\" (UniqueName: \"kubernetes.io/projected/d56b1153-6a1c-45c5-bd51-6e8b40ef58ec-kube-api-access-qlj48\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg\" (UID: \"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:25.888750 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:25.888721 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:26.202808 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:26.202719 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg"] Oct 30 12:12:26.207686 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:12:26.207652 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56b1153_6a1c_45c5_bd51_6e8b40ef58ec.slice/crio-3dfb52f118baf49e81bac278340015cfd8d9eb4ce4111fb490f28d0cacca672b WatchSource:0}: Error finding container 3dfb52f118baf49e81bac278340015cfd8d9eb4ce4111fb490f28d0cacca672b: Status 404 returned error can't find the container with id 3dfb52f118baf49e81bac278340015cfd8d9eb4ce4111fb490f28d0cacca672b Oct 30 12:12:26.873937 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:26.873903 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" event={"ID":"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec","Type":"ContainerStarted","Data":"0ea365a22bc6e544611339f4e2357461986bc85dfebe279e1ad0bd374095b4f3"} Oct 30 12:12:26.874343 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:26.873965 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" event={"ID":"d56b1153-6a1c-45c5-bd51-6e8b40ef58ec","Type":"ContainerStarted","Data":"3dfb52f118baf49e81bac278340015cfd8d9eb4ce4111fb490f28d0cacca672b"} Oct 30 12:12:26.889282 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:26.889261 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:27.354758 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.354659 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" podStartSLOduration=2.146246246 podStartE2EDuration="2.354644957s" totalImagesPullingTime="208.398711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:26.91850656 +0000 UTC m=+1100.028349444" watchObservedRunningTime="2026-10-30 12:12:27.354644957 +0000 UTC m=+1100.464487903" Oct 30 12:12:27.354946 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.354927 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-l4qzc"] Oct 30 12:12:27.358554 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.358534 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.373409 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.373388 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 12:12:27.385780 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.385748 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcfrj\" (UniqueName: \"kubernetes.io/projected/da4159b0-d9f6-4cd8-9138-e104f230d251-kube-api-access-tcfrj\") pod \"echo-simulator-kserve-64b6f8f96c-l4qzc\" (UID: \"da4159b0-d9f6-4cd8-9138-e104f230d251\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.385918 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.385866 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4159b0-d9f6-4cd8-9138-e104f230d251-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-l4qzc\" (UID: \"da4159b0-d9f6-4cd8-9138-e104f230d251\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.385969 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.385918 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4159b0-d9f6-4cd8-9138-e104f230d251-home\") pod \"echo-simulator-kserve-64b6f8f96c-l4qzc\" (UID: \"da4159b0-d9f6-4cd8-9138-e104f230d251\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.386012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.385980 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4159b0-d9f6-4cd8-9138-e104f230d251-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-l4qzc\" (UID: \"da4159b0-d9f6-4cd8-9138-e104f230d251\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.386012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.386001 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4159b0-d9f6-4cd8-9138-e104f230d251-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-l4qzc\" (UID: \"da4159b0-d9f6-4cd8-9138-e104f230d251\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.386087 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.386039 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4159b0-d9f6-4cd8-9138-e104f230d251-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-l4qzc\" (UID: \"da4159b0-d9f6-4cd8-9138-e104f230d251\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.388237 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.388212 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-l4qzc"] Oct 30 12:12:27.435602 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.435574 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:12:27.486801 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.486782 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4159b0-d9f6-4cd8-9138-e104f230d251-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-l4qzc\" (UID: \"da4159b0-d9f6-4cd8-9138-e104f230d251\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.486919 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.486814 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4159b0-d9f6-4cd8-9138-e104f230d251-home\") pod \"echo-simulator-kserve-64b6f8f96c-l4qzc\" (UID: \"da4159b0-d9f6-4cd8-9138-e104f230d251\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.486919 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.486882 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4159b0-d9f6-4cd8-9138-e104f230d251-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-l4qzc\" (UID: \"da4159b0-d9f6-4cd8-9138-e104f230d251\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.487003 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.486918 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4159b0-d9f6-4cd8-9138-e104f230d251-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-l4qzc\" (UID: \"da4159b0-d9f6-4cd8-9138-e104f230d251\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.487003 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.486971 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4159b0-d9f6-4cd8-9138-e104f230d251-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-l4qzc\" (UID: \"da4159b0-d9f6-4cd8-9138-e104f230d251\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.487095 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.487029 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcfrj\" (UniqueName: \"kubernetes.io/projected/da4159b0-d9f6-4cd8-9138-e104f230d251-kube-api-access-tcfrj\") pod \"echo-simulator-kserve-64b6f8f96c-l4qzc\" (UID: \"da4159b0-d9f6-4cd8-9138-e104f230d251\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.487242 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.487226 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4159b0-d9f6-4cd8-9138-e104f230d251-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-l4qzc\" (UID: \"da4159b0-d9f6-4cd8-9138-e104f230d251\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.487364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.487337 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4159b0-d9f6-4cd8-9138-e104f230d251-home\") pod \"echo-simulator-kserve-64b6f8f96c-l4qzc\" (UID: \"da4159b0-d9f6-4cd8-9138-e104f230d251\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.487486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.487411 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4159b0-d9f6-4cd8-9138-e104f230d251-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-l4qzc\" (UID: \"da4159b0-d9f6-4cd8-9138-e104f230d251\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.489401 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.489385 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4159b0-d9f6-4cd8-9138-e104f230d251-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-l4qzc\" (UID: \"da4159b0-d9f6-4cd8-9138-e104f230d251\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.489565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.489550 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4159b0-d9f6-4cd8-9138-e104f230d251-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-l4qzc\" (UID: \"da4159b0-d9f6-4cd8-9138-e104f230d251\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.509354 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.509328 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcfrj\" (UniqueName: \"kubernetes.io/projected/da4159b0-d9f6-4cd8-9138-e104f230d251-kube-api-access-tcfrj\") pod \"echo-simulator-kserve-64b6f8f96c-l4qzc\" (UID: \"da4159b0-d9f6-4cd8-9138-e104f230d251\") " pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.675652 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.675613 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:27.975932 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:27.975908 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-l4qzc"] Oct 30 12:12:27.977913 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:12:27.977881 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4159b0_d9f6_4cd8_9138_e104f230d251.slice/crio-0d82453912ef2506caa87378cfd672f0c7d4a2d1a4b39fd3d0275d557345f3fd WatchSource:0}: Error finding container 0d82453912ef2506caa87378cfd672f0c7d4a2d1a4b39fd3d0275d557345f3fd: Status 404 returned error can't find the container with id 0d82453912ef2506caa87378cfd672f0c7d4a2d1a4b39fd3d0275d557345f3fd Oct 30 12:12:28.886139 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:28.886103 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" event={"ID":"da4159b0-d9f6-4cd8-9138-e104f230d251","Type":"ContainerStarted","Data":"0d82453912ef2506caa87378cfd672f0c7d4a2d1a4b39fd3d0275d557345f3fd"} Oct 30 12:12:29.926569 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:29.926528 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" event={"ID":"da4159b0-d9f6-4cd8-9138-e104f230d251","Type":"ContainerStarted","Data":"e5a57e094f757e553909e1d1e6c034c7803b38624db094ff95bdbeb5fa2078e2"} Oct 30 12:12:29.977179 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:29.977127 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" podStartSLOduration=1.526639708 podStartE2EDuration="2.977111301s" totalImagesPullingTime="1.450471593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:29.975935011 +0000 UTC m=+1103.085777893" watchObservedRunningTime="2026-10-30 12:12:29.977111301 +0000 UTC m=+1103.086954183" Oct 30 12:12:30.676489 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.675336 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9"] Oct 30 12:12:30.680570 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.679865 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9"] Oct 30 12:12:30.680570 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.679915 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:30.680570 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.680015 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:30.747291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.736960 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 12:12:30.757526 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.757277 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be353ed4-7110-45d5-a5f6-489cca57fdcb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9\" (UID: \"be353ed4-7110-45d5-a5f6-489cca57fdcb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:30.757526 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.757329 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be353ed4-7110-45d5-a5f6-489cca57fdcb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9\" (UID: \"be353ed4-7110-45d5-a5f6-489cca57fdcb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:30.757526 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.757362 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4xhz\" (UniqueName: \"kubernetes.io/projected/be353ed4-7110-45d5-a5f6-489cca57fdcb-kube-api-access-c4xhz\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9\" (UID: \"be353ed4-7110-45d5-a5f6-489cca57fdcb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:30.757526 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.757413 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be353ed4-7110-45d5-a5f6-489cca57fdcb-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9\" (UID: \"be353ed4-7110-45d5-a5f6-489cca57fdcb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:30.757526 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.757464 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be353ed4-7110-45d5-a5f6-489cca57fdcb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9\" (UID: \"be353ed4-7110-45d5-a5f6-489cca57fdcb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:30.757952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.757547 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be353ed4-7110-45d5-a5f6-489cca57fdcb-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9\" (UID: \"be353ed4-7110-45d5-a5f6-489cca57fdcb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:30.762657 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.759044 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:12:30.865275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.864798 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be353ed4-7110-45d5-a5f6-489cca57fdcb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9\" (UID: \"be353ed4-7110-45d5-a5f6-489cca57fdcb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:30.865275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.864846 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be353ed4-7110-45d5-a5f6-489cca57fdcb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9\" (UID: \"be353ed4-7110-45d5-a5f6-489cca57fdcb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:30.865275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.864881 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4xhz\" (UniqueName: \"kubernetes.io/projected/be353ed4-7110-45d5-a5f6-489cca57fdcb-kube-api-access-c4xhz\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9\" (UID: \"be353ed4-7110-45d5-a5f6-489cca57fdcb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:30.865275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.864936 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be353ed4-7110-45d5-a5f6-489cca57fdcb-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9\" (UID: \"be353ed4-7110-45d5-a5f6-489cca57fdcb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:30.865275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.864992 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be353ed4-7110-45d5-a5f6-489cca57fdcb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9\" (UID: \"be353ed4-7110-45d5-a5f6-489cca57fdcb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:30.865275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.865080 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be353ed4-7110-45d5-a5f6-489cca57fdcb-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9\" (UID: \"be353ed4-7110-45d5-a5f6-489cca57fdcb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:30.868324 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.866928 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be353ed4-7110-45d5-a5f6-489cca57fdcb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9\" (UID: \"be353ed4-7110-45d5-a5f6-489cca57fdcb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:30.868324 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.867791 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be353ed4-7110-45d5-a5f6-489cca57fdcb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9\" (UID: \"be353ed4-7110-45d5-a5f6-489cca57fdcb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:30.868324 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.867844 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be353ed4-7110-45d5-a5f6-489cca57fdcb-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9\" (UID: \"be353ed4-7110-45d5-a5f6-489cca57fdcb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:30.869290 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.868818 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be353ed4-7110-45d5-a5f6-489cca57fdcb-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9\" (UID: \"be353ed4-7110-45d5-a5f6-489cca57fdcb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:30.870784 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.870762 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be353ed4-7110-45d5-a5f6-489cca57fdcb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9\" (UID: \"be353ed4-7110-45d5-a5f6-489cca57fdcb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:30.896273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:30.890508 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4xhz\" (UniqueName: \"kubernetes.io/projected/be353ed4-7110-45d5-a5f6-489cca57fdcb-kube-api-access-c4xhz\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9\" (UID: \"be353ed4-7110-45d5-a5f6-489cca57fdcb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:31.074330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:31.073983 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:31.393221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:31.393159 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9"] Oct 30 12:12:31.395931 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:12:31.395900 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe353ed4_7110_45d5_a5f6_489cca57fdcb.slice/crio-29d5c418b3eab677005b9d67d78a0a005f8b0284c564804e274940e7f6858df3 WatchSource:0}: Error finding container 29d5c418b3eab677005b9d67d78a0a005f8b0284c564804e274940e7f6858df3: Status 404 returned error can't find the container with id 29d5c418b3eab677005b9d67d78a0a005f8b0284c564804e274940e7f6858df3 Oct 30 12:12:31.952519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:31.952483 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" event={"ID":"be353ed4-7110-45d5-a5f6-489cca57fdcb","Type":"ContainerStarted","Data":"b83027b9f4670c2fa92238315276cde8f5471c81fa36c8a8b8694cf56ed55b3d"} Oct 30 12:12:31.952676 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:31.952544 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" event={"ID":"be353ed4-7110-45d5-a5f6-489cca57fdcb","Type":"ContainerStarted","Data":"29d5c418b3eab677005b9d67d78a0a005f8b0284c564804e274940e7f6858df3"} Oct 30 12:12:31.989961 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:31.989936 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:32.006189 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:32.006168 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:32.047388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:32.047343 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" podStartSLOduration=1.862590971 podStartE2EDuration="2.047330271s" totalImagesPullingTime="184.7393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:31.997753075 +0000 UTC m=+1105.107595958" watchObservedRunningTime="2026-10-30 12:12:32.047330271 +0000 UTC m=+1105.157173152" Oct 30 12:12:32.074613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:32.074591 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:32.351575 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:32.351507 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:12:32.967979 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:32.967951 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wvt6m" Oct 30 12:12:35.875100 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:12:35.875064 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:12:35.889502 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:35.889472 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:35.929275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:35.928358 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:35.987579 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:35.987553 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg" Oct 30 12:12:37.676191 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:37.676158 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:37.689121 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:37.689093 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:37.997789 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:37.997725 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-l4qzc" Oct 30 12:12:41.075403 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:41.075366 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:41.087963 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:41.087942 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:42.018657 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:42.018628 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9" Oct 30 12:12:51.739121 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:51.738971 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:12:51.867755 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:51.867726 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:12:53.403570 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:53.403542 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:12:53.554243 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:53.554199 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:12:55.706426 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:12:55.706382 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:05.814113 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:13:05.814067 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:13:21.376630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:21.376585 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:21.514058 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:21.514026 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:21.642069 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:21.641976 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:21.871997 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:21.871968 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:22.159502 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.159468 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:22.292326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.292292 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:22.420675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.420590 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:22.481193 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.481161 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5"] Oct 30 12:13:22.488798 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.488770 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.507790 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.507763 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5"] Oct 30 12:13:22.545600 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.545570 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:22.591467 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.591444 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-307283d2-gw-openshift-default-dockercfg-lg7xk\"" Oct 30 12:13:22.603015 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.602979 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-workload-certs\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.603207 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.603022 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-envoy\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.603207 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.603163 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-token\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.603340 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.603221 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-workload-socket\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.603380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.603337 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-podinfo\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.603464 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.603401 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dj8p\" (UniqueName: \"kubernetes.io/projected/4bc64fed-721b-4b8d-91bd-4ca8331409c1-kube-api-access-8dj8p\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.603464 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.603444 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-credential-socket\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.603566 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.603519 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-data\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.603602 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.603585 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istiod-ca-cert\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.705166 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.705077 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-data\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.705166 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.705133 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istiod-ca-cert\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.705166 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.705158 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-workload-certs\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.705564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.705175 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-envoy\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.705564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.705210 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-token\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.705564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.705292 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-workload-socket\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.705564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.705421 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-podinfo\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.705564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.705491 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dj8p\" (UniqueName: \"kubernetes.io/projected/4bc64fed-721b-4b8d-91bd-4ca8331409c1-kube-api-access-8dj8p\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.705564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.705538 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-credential-socket\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.705877 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.705606 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-data\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.705877 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.705628 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-workload-certs\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.705877 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.705718 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-workload-socket\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.706037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.705874 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-credential-socket\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.706037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.705992 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istiod-ca-cert\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.707773 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.707750 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-envoy\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.708028 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.708009 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-podinfo\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.728664 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.728629 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-token\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.728807 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.728737 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dj8p\" (UniqueName: \"kubernetes.io/projected/4bc64fed-721b-4b8d-91bd-4ca8331409c1-kube-api-access-8dj8p\") pod \"e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:22.741540 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.741509 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:22.803747 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:22.803714 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:23.132693 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.132668 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5"] Oct 30 12:13:23.134949 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:13:23.134922 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc64fed_721b_4b8d_91bd_4ca8331409c1.slice/crio-e3e33fffaf57c2da2bcb61865fadf947f782bcfe9d2e78621b365faa5fd73783 WatchSource:0}: Error finding container e3e33fffaf57c2da2bcb61865fadf947f782bcfe9d2e78621b365faa5fd73783: Status 404 returned error can't find the container with id e3e33fffaf57c2da2bcb61865fadf947f782bcfe9d2e78621b365faa5fd73783 Oct 30 12:13:23.136946 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.136914 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:13:23.137017 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.136976 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:13:23.137017 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.137003 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:13:23.212932 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.212905 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" event={"ID":"4bc64fed-721b-4b8d-91bd-4ca8331409c1","Type":"ContainerStarted","Data":"e621ac718a5a1e3696a2553f333583f5dccb2cc4a95403e25264527e09e7e5a6"} Oct 30 12:13:23.213082 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.212961 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" event={"ID":"4bc64fed-721b-4b8d-91bd-4ca8331409c1","Type":"ContainerStarted","Data":"e3e33fffaf57c2da2bcb61865fadf947f782bcfe9d2e78621b365faa5fd73783"} Oct 30 12:13:23.259964 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.259915 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" podStartSLOduration=1.259902335 podStartE2EDuration="1.259902335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:23.257883805 +0000 UTC m=+1156.367726714" watchObservedRunningTime="2026-10-30 12:13:23.259902335 +0000 UTC m=+1156.369745215" Oct 30 12:13:23.320411 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.320370 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:23.746196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.746163 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg"] Oct 30 12:13:23.749903 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.749880 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.765142 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.765121 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e636da41-openshift-default-dockercfg-k6sm2\"" Oct 30 12:13:23.775188 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.775150 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg"] Oct 30 12:13:23.804831 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.804803 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:23.816120 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.816088 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-workload-certs\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.816306 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.816222 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-workload-socket\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.816392 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.816302 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-podinfo\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.816392 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.816336 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-envoy\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.816484 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.816411 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-data\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.816524 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.816478 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-credential-socket\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.816568 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.816556 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjq7j\" (UniqueName: \"kubernetes.io/projected/327fe8ba-262d-4b2d-9948-a406ffc77e1c-kube-api-access-cjq7j\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.816604 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.816578 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-token\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.816604 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.816596 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istiod-ca-cert\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.844506 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.844463 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:23.917443 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.917411 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-workload-certs\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.917630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.917471 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-workload-socket\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.917630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.917506 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-podinfo\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.917630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.917527 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-envoy\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.917630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.917551 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-data\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.917630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.917572 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-credential-socket\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.917630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.917615 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjq7j\" (UniqueName: \"kubernetes.io/projected/327fe8ba-262d-4b2d-9948-a406ffc77e1c-kube-api-access-cjq7j\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.917969 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.917652 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-token\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.917969 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.917680 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istiod-ca-cert\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.917969 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.917866 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-workload-socket\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.917969 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.917903 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-workload-certs\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.918189 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.918166 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-credential-socket\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.918238 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.918190 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-data\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.918405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.918388 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istiod-ca-cert\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.920127 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.920109 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-podinfo\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.920732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.920717 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-envoy\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.945279 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.945215 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjq7j\" (UniqueName: \"kubernetes.io/projected/327fe8ba-262d-4b2d-9948-a406ffc77e1c-kube-api-access-cjq7j\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.945561 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.945541 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-token\") pod \"e2e-ait-e636da41-openshift-default-99df89b57-62xrg\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:23.965802 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:23.965751 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:24.065584 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:24.065494 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:24.383026 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:24.382998 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg"] Oct 30 12:13:24.385007 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:13:24.384982 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327fe8ba_262d_4b2d_9948_a406ffc77e1c.slice/crio-f2e604bfee293e226e362702b2a848aaec9d901ac6d766f1bdf59e6760853877 WatchSource:0}: Error finding container f2e604bfee293e226e362702b2a848aaec9d901ac6d766f1bdf59e6760853877: Status 404 returned error can't find the container with id f2e604bfee293e226e362702b2a848aaec9d901ac6d766f1bdf59e6760853877 Oct 30 12:13:24.387148 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:24.387116 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:13:24.387218 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:24.387181 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:13:24.387218 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:24.387208 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:13:24.538621 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:24.538592 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:24.680610 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:24.680523 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:24.809764 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:24.809731 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:25.163899 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:25.163865 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:25.224429 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:25.224391 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" event={"ID":"327fe8ba-262d-4b2d-9948-a406ffc77e1c","Type":"ContainerStarted","Data":"34cca965db48c5c7ed3ba167e35a8bb6bdab3852e96f077903689ed06f6cbd58"} Oct 30 12:13:25.224621 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:25.224452 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" event={"ID":"327fe8ba-262d-4b2d-9948-a406ffc77e1c","Type":"ContainerStarted","Data":"f2e604bfee293e226e362702b2a848aaec9d901ac6d766f1bdf59e6760853877"} Oct 30 12:13:25.224760 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:25.224730 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:25.225724 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:25.225708 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:25.272084 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:25.272037 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" podStartSLOduration=2.272023055 podStartE2EDuration="2.272023055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:25.27041582 +0000 UTC m=+1158.380258700" watchObservedRunningTime="2026-10-30 12:13:25.272023055 +0000 UTC m=+1158.381865936" Oct 30 12:13:25.281909 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:25.281885 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:26.042585 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:26.042557 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:26.066426 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:26.066386 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:26.071808 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:26.071787 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:26.229416 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:26.229385 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:26.230057 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:26.230041 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:26.315545 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:26.315469 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:27.731760 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.731724 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:27.751798 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.751768 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg"] Oct 30 12:13:27.757453 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.757433 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" Oct 30 12:13:27.782303 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.782270 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg"] Oct 30 12:13:27.789313 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.789287 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-307283d2-serving-cert\"" Oct 30 12:13:27.849106 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.849066 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:27.860777 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.860750 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3f923a9-c8d9-40ad-8786-c6058f639fb0-maas-api-tls\") pod \"maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg\" (UID: \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" Oct 30 12:13:27.860910 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.860881 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frpwh\" (UniqueName: \"kubernetes.io/projected/b3f923a9-c8d9-40ad-8786-c6058f639fb0-kube-api-access-frpwh\") pod \"maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg\" (UID: \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" Oct 30 12:13:27.861046 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.861025 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f923a9-c8d9-40ad-8786-c6058f639fb0-openshift-service-ca\") pod \"maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg\" (UID: \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" Oct 30 12:13:27.861116 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.861099 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3f923a9-c8d9-40ad-8786-c6058f639fb0-metrics-certs\") pod \"maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg\" (UID: \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" Oct 30 12:13:27.962068 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.962032 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3f923a9-c8d9-40ad-8786-c6058f639fb0-maas-api-tls\") pod \"maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg\" (UID: \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" Oct 30 12:13:27.962214 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.962100 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frpwh\" (UniqueName: \"kubernetes.io/projected/b3f923a9-c8d9-40ad-8786-c6058f639fb0-kube-api-access-frpwh\") pod \"maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg\" (UID: \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" Oct 30 12:13:27.962214 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.962199 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f923a9-c8d9-40ad-8786-c6058f639fb0-openshift-service-ca\") pod \"maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg\" (UID: \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" Oct 30 12:13:27.962356 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.962269 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3f923a9-c8d9-40ad-8786-c6058f639fb0-metrics-certs\") pod \"maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg\" (UID: \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" Oct 30 12:13:27.962900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.962824 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f923a9-c8d9-40ad-8786-c6058f639fb0-openshift-service-ca\") pod \"maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg\" (UID: \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" Oct 30 12:13:27.964725 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.964697 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3f923a9-c8d9-40ad-8786-c6058f639fb0-maas-api-tls\") pod \"maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg\" (UID: \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" Oct 30 12:13:27.964860 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.964842 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3f923a9-c8d9-40ad-8786-c6058f639fb0-metrics-certs\") pod \"maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg\" (UID: \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" Oct 30 12:13:27.973201 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.973174 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:27.984842 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:27.984784 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frpwh\" (UniqueName: \"kubernetes.io/projected/b3f923a9-c8d9-40ad-8786-c6058f639fb0-kube-api-access-frpwh\") pod \"maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg\" (UID: \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" Oct 30 12:13:28.077408 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:28.077380 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" Oct 30 12:13:28.084651 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:28.084624 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:28.358450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:28.358379 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:28.386020 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:28.385995 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg"] Oct 30 12:13:28.387606 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:13:28.387582 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f923a9_c8d9_40ad_8786_c6058f639fb0.slice/crio-ad74de5fe54f4c0bd601d72aa7e161ce38ef8ea5fa5514ba393b869a215915e2 WatchSource:0}: Error finding container ad74de5fe54f4c0bd601d72aa7e161ce38ef8ea5fa5514ba393b869a215915e2: Status 404 returned error can't find the container with id ad74de5fe54f4c0bd601d72aa7e161ce38ef8ea5fa5514ba393b869a215915e2 Oct 30 12:13:28.481734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:28.481704 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:29.245669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.245625 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" event={"ID":"b3f923a9-c8d9-40ad-8786-c6058f639fb0","Type":"ContainerStarted","Data":"9ccf40639de88ad19f51f641d2b3337e73d508bf00e005388b073ee92e024a25"} Oct 30 12:13:29.246027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.245687 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" event={"ID":"b3f923a9-c8d9-40ad-8786-c6058f639fb0","Type":"ContainerStarted","Data":"ad74de5fe54f4c0bd601d72aa7e161ce38ef8ea5fa5514ba393b869a215915e2"} Oct 30 12:13:29.246027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.245729 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" Oct 30 12:13:29.282919 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.282863 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:29.288799 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.288757 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" podStartSLOduration=1.813131397 podStartE2EDuration="2.288744597s" totalImagesPullingTime="475.6132ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:29.288443767 +0000 UTC m=+1162.398286639" watchObservedRunningTime="2026-10-30 12:13:29.288744597 +0000 UTC m=+1162.398587477" Oct 30 12:13:29.398298 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.398269 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:29.521144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.521073 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:29.637399 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.637371 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:29.715946 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.715916 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j"] Oct 30 12:13:29.719697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.719681 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:13:29.737153 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.737131 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1c85bd66-serving-cert\"" Oct 30 12:13:29.745198 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.745175 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j"] Oct 30 12:13:29.791376 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.791313 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjxsg\" (UniqueName: \"kubernetes.io/projected/d871951f-2819-4c28-880d-ffc4460b66da-kube-api-access-hjxsg\") pod \"maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j\" (UID: \"d871951f-2819-4c28-880d-ffc4460b66da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:13:29.791376 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.791366 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d871951f-2819-4c28-880d-ffc4460b66da-metrics-certs\") pod \"maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j\" (UID: \"d871951f-2819-4c28-880d-ffc4460b66da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:13:29.791531 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.791432 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d871951f-2819-4c28-880d-ffc4460b66da-maas-api-tls\") pod \"maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j\" (UID: \"d871951f-2819-4c28-880d-ffc4460b66da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:13:29.791531 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.791509 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d871951f-2819-4c28-880d-ffc4460b66da-openshift-service-ca\") pod \"maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j\" (UID: \"d871951f-2819-4c28-880d-ffc4460b66da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:13:29.892003 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.891976 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjxsg\" (UniqueName: \"kubernetes.io/projected/d871951f-2819-4c28-880d-ffc4460b66da-kube-api-access-hjxsg\") pod \"maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j\" (UID: \"d871951f-2819-4c28-880d-ffc4460b66da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:13:29.892131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.892023 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d871951f-2819-4c28-880d-ffc4460b66da-metrics-certs\") pod \"maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j\" (UID: \"d871951f-2819-4c28-880d-ffc4460b66da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:13:29.892131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.892042 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d871951f-2819-4c28-880d-ffc4460b66da-maas-api-tls\") pod \"maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j\" (UID: \"d871951f-2819-4c28-880d-ffc4460b66da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:13:29.892131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.892060 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d871951f-2819-4c28-880d-ffc4460b66da-openshift-service-ca\") pod \"maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j\" (UID: \"d871951f-2819-4c28-880d-ffc4460b66da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:13:29.892227 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:13:29.892134 4413 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-serving-cert: secret "maas-api-e2e-ait-1c85bd66-serving-cert" not found Oct 30 12:13:29.892227 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:13:29.892199 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d871951f-2819-4c28-880d-ffc4460b66da-maas-api-tls podName:d871951f-2819-4c28-880d-ffc4460b66da nodeName:}" failed. No retries permitted until 2026-10-30 12:13:30.392181015 +0000 UTC m=+1163.502023891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d871951f-2819-4c28-880d-ffc4460b66da-maas-api-tls") pod "maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" (UID: "d871951f-2819-4c28-880d-ffc4460b66da") : secret "maas-api-e2e-ait-1c85bd66-serving-cert" not found Oct 30 12:13:29.892639 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.892622 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d871951f-2819-4c28-880d-ffc4460b66da-openshift-service-ca\") pod \"maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j\" (UID: \"d871951f-2819-4c28-880d-ffc4460b66da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:13:29.894690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.894665 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d871951f-2819-4c28-880d-ffc4460b66da-metrics-certs\") pod \"maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j\" (UID: \"d871951f-2819-4c28-880d-ffc4460b66da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:13:29.933870 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.933811 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjxsg\" (UniqueName: \"kubernetes.io/projected/d871951f-2819-4c28-880d-ffc4460b66da-kube-api-access-hjxsg\") pod \"maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j\" (UID: \"d871951f-2819-4c28-880d-ffc4460b66da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:13:29.952498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:29.952471 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:30.079164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:30.078696 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:30.207852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:30.207667 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:30.397415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:30.397387 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d871951f-2819-4c28-880d-ffc4460b66da-maas-api-tls\") pod \"maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j\" (UID: \"d871951f-2819-4c28-880d-ffc4460b66da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:13:30.400074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:30.400049 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d871951f-2819-4c28-880d-ffc4460b66da-maas-api-tls\") pod \"maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j\" (UID: \"d871951f-2819-4c28-880d-ffc4460b66da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:13:30.621190 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:30.621160 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:30.633131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:30.633107 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:13:30.932700 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:30.932680 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j"] Oct 30 12:13:30.937298 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:13:30.934653 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd871951f_2819_4c28_880d_ffc4460b66da.slice/crio-39bf1741eedacc4016ba057193adc0af27d4366f53dde7fcd27ab851c2296c37 WatchSource:0}: Error finding container 39bf1741eedacc4016ba057193adc0af27d4366f53dde7fcd27ab851c2296c37: Status 404 returned error can't find the container with id 39bf1741eedacc4016ba057193adc0af27d4366f53dde7fcd27ab851c2296c37 Oct 30 12:13:31.016561 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:31.016529 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:31.133909 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:31.133871 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:31.189721 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:31.189651 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8"] Oct 30 12:13:31.194490 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:31.194469 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8" Oct 30 12:13:31.212682 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:31.212658 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e636da41-dockercfg-9fs5l\"" Oct 30 12:13:31.214893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:31.214873 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8"] Oct 30 12:13:31.242221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:31.242188 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e636da41\"" Oct 30 12:13:31.258889 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:31.258863 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" event={"ID":"d871951f-2819-4c28-880d-ffc4460b66da","Type":"ContainerStarted","Data":"39bf1741eedacc4016ba057193adc0af27d4366f53dde7fcd27ab851c2296c37"} Oct 30 12:13:31.305597 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:31.305567 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj2xz\" (UniqueName: \"kubernetes.io/projected/d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89-kube-api-access-vj2xz\") pod \"payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8\" (UID: \"d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8" Oct 30 12:13:31.305777 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:31.305634 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8\" (UID: \"d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8" Oct 30 12:13:31.406300 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:31.406276 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj2xz\" (UniqueName: \"kubernetes.io/projected/d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89-kube-api-access-vj2xz\") pod \"payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8\" (UID: \"d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8" Oct 30 12:13:31.406619 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:31.406353 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8\" (UID: \"d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8" Oct 30 12:13:31.407131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:31.407111 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8\" (UID: \"d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8" Oct 30 12:13:31.428886 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:31.428852 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj2xz\" (UniqueName: \"kubernetes.io/projected/d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89-kube-api-access-vj2xz\") pod \"payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8\" (UID: \"d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8" Oct 30 12:13:31.507271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:31.507187 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8" Oct 30 12:13:31.817800 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:31.817778 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8"] Oct 30 12:13:31.819699 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:13:31.819672 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5aef5d2_76c1_4fec_a9c4_5aff1da8ec89.slice/crio-0b5f43c3187016a44cfebaca73c3ae16f62bf5c3b7f10f2fe48cf15b97b57237 WatchSource:0}: Error finding container 0b5f43c3187016a44cfebaca73c3ae16f62bf5c3b7f10f2fe48cf15b97b57237: Status 404 returned error can't find the container with id 0b5f43c3187016a44cfebaca73c3ae16f62bf5c3b7f10f2fe48cf15b97b57237 Oct 30 12:13:31.855411 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:31.855381 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:32.270780 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:32.270734 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" event={"ID":"d871951f-2819-4c28-880d-ffc4460b66da","Type":"ContainerStarted","Data":"bb32c2c7631bfb35010a8bc0a1f31b08c30e87c8712fbe7f5abc9113f5c7df82"} Oct 30 12:13:32.270954 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:32.270802 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:13:32.272817 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:32.272785 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8" event={"ID":"d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89","Type":"ContainerStarted","Data":"0b5f43c3187016a44cfebaca73c3ae16f62bf5c3b7f10f2fe48cf15b97b57237"} Oct 30 12:13:32.334895 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:32.334845 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" podStartSLOduration=2.908940844 podStartE2EDuration="3.334829885s" totalImagesPullingTime="425.889041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:32.33365916 +0000 UTC m=+1165.443502046" watchObservedRunningTime="2026-10-30 12:13:32.334829885 +0000 UTC m=+1165.444672794" Oct 30 12:13:32.957548 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:32.957163 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:33.075687 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.075656 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:33.725330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.724330 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s"] Oct 30 12:13:33.733624 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.733602 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" Oct 30 12:13:33.750796 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.750735 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s"] Oct 30 12:13:33.816615 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.816580 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-80143318-serving-cert\"" Oct 30 12:13:33.817994 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.817913 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:33.835861 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.835832 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0bf1f48-c28e-42d1-9802-9a73475c9946-metrics-certs\") pod \"maas-api-e2e-ait-80143318-77c9644786-g5m2s\" (UID: \"f0bf1f48-c28e-42d1-9802-9a73475c9946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" Oct 30 12:13:33.836016 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.835958 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42lkg\" (UniqueName: \"kubernetes.io/projected/f0bf1f48-c28e-42d1-9802-9a73475c9946-kube-api-access-42lkg\") pod \"maas-api-e2e-ait-80143318-77c9644786-g5m2s\" (UID: \"f0bf1f48-c28e-42d1-9802-9a73475c9946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" Oct 30 12:13:33.836016 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.836001 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bf1f48-c28e-42d1-9802-9a73475c9946-openshift-service-ca\") pod \"maas-api-e2e-ait-80143318-77c9644786-g5m2s\" (UID: \"f0bf1f48-c28e-42d1-9802-9a73475c9946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" Oct 30 12:13:33.836099 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.836055 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0bf1f48-c28e-42d1-9802-9a73475c9946-maas-api-tls\") pod \"maas-api-e2e-ait-80143318-77c9644786-g5m2s\" (UID: \"f0bf1f48-c28e-42d1-9802-9a73475c9946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" Oct 30 12:13:33.938039 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.937419 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0bf1f48-c28e-42d1-9802-9a73475c9946-metrics-certs\") pod \"maas-api-e2e-ait-80143318-77c9644786-g5m2s\" (UID: \"f0bf1f48-c28e-42d1-9802-9a73475c9946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" Oct 30 12:13:33.938039 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.937501 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42lkg\" (UniqueName: \"kubernetes.io/projected/f0bf1f48-c28e-42d1-9802-9a73475c9946-kube-api-access-42lkg\") pod \"maas-api-e2e-ait-80143318-77c9644786-g5m2s\" (UID: \"f0bf1f48-c28e-42d1-9802-9a73475c9946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" Oct 30 12:13:33.938039 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.937537 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bf1f48-c28e-42d1-9802-9a73475c9946-openshift-service-ca\") pod \"maas-api-e2e-ait-80143318-77c9644786-g5m2s\" (UID: \"f0bf1f48-c28e-42d1-9802-9a73475c9946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" Oct 30 12:13:33.938039 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.937602 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0bf1f48-c28e-42d1-9802-9a73475c9946-maas-api-tls\") pod \"maas-api-e2e-ait-80143318-77c9644786-g5m2s\" (UID: \"f0bf1f48-c28e-42d1-9802-9a73475c9946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" Oct 30 12:13:33.938468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.938441 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bf1f48-c28e-42d1-9802-9a73475c9946-openshift-service-ca\") pod \"maas-api-e2e-ait-80143318-77c9644786-g5m2s\" (UID: \"f0bf1f48-c28e-42d1-9802-9a73475c9946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" Oct 30 12:13:33.944281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.941582 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0bf1f48-c28e-42d1-9802-9a73475c9946-metrics-certs\") pod \"maas-api-e2e-ait-80143318-77c9644786-g5m2s\" (UID: \"f0bf1f48-c28e-42d1-9802-9a73475c9946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" Oct 30 12:13:33.944281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.944132 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0bf1f48-c28e-42d1-9802-9a73475c9946-maas-api-tls\") pod \"maas-api-e2e-ait-80143318-77c9644786-g5m2s\" (UID: \"f0bf1f48-c28e-42d1-9802-9a73475c9946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" Oct 30 12:13:33.951480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.951426 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:33.963148 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:33.963120 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42lkg\" (UniqueName: \"kubernetes.io/projected/f0bf1f48-c28e-42d1-9802-9a73475c9946-kube-api-access-42lkg\") pod \"maas-api-e2e-ait-80143318-77c9644786-g5m2s\" (UID: \"f0bf1f48-c28e-42d1-9802-9a73475c9946\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" Oct 30 12:13:34.052055 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:34.051982 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" Oct 30 12:13:34.069336 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:34.069310 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:34.412277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:34.392817 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s"] Oct 30 12:13:34.419804 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:13:34.419775 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bf1f48_c28e_42d1_9802_9a73475c9946.slice/crio-0c16a48437a7776a9f58ed3e7d1856e9c9047dce73565a0b2f84897bafdbd586 WatchSource:0}: Error finding container 0c16a48437a7776a9f58ed3e7d1856e9c9047dce73565a0b2f84897bafdbd586: Status 404 returned error can't find the container with id 0c16a48437a7776a9f58ed3e7d1856e9c9047dce73565a0b2f84897bafdbd586 Oct 30 12:13:34.773320 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:34.773288 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:34.920295 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:34.918453 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:35.255690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:35.255664 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" Oct 30 12:13:35.323807 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:35.323035 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" event={"ID":"f0bf1f48-c28e-42d1-9802-9a73475c9946","Type":"ContainerStarted","Data":"7d6098ba8d43c1ead6a2d69b08f8b59427adccbc5335b50046c466842d7e2677"} Oct 30 12:13:35.323807 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:35.323094 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" event={"ID":"f0bf1f48-c28e-42d1-9802-9a73475c9946","Type":"ContainerStarted","Data":"0c16a48437a7776a9f58ed3e7d1856e9c9047dce73565a0b2f84897bafdbd586"} Oct 30 12:13:35.323807 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:35.323367 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" Oct 30 12:13:35.325127 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:35.325104 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8" event={"ID":"d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89","Type":"ContainerStarted","Data":"b26cd6747a335703913bba76db60596f515d5a5738f19c235e1de222fc1e4823"} Oct 30 12:13:35.325420 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:35.325407 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8" Oct 30 12:13:35.364784 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:35.364740 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" podStartSLOduration=2.036547005 podStartE2EDuration="2.364726973s" totalImagesPullingTime="328.179968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:35.363565518 +0000 UTC m=+1168.473408400" watchObservedRunningTime="2026-10-30 12:13:35.364726973 +0000 UTC m=+1168.474569866" Oct 30 12:13:35.416282 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:35.416189 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8" podStartSLOduration=1.69740135 podStartE2EDuration="4.416176921s" totalImagesPullingTime="2.718775571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:35.415794173 +0000 UTC m=+1168.525637054" watchObservedRunningTime="2026-10-30 12:13:35.416176921 +0000 UTC m=+1168.526019802" Oct 30 12:13:35.816067 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:35.816011 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:35.840450 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:13:35.840417 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:13:37.313277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.312788 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w"] Oct 30 12:13:37.319187 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.319163 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" Oct 30 12:13:37.335603 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.335552 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w"] Oct 30 12:13:37.346851 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.346828 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e636da41-serving-cert\"" Oct 30 12:13:37.418604 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.418571 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:37.477468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.477439 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19995406-ce1f-4562-9de5-5566315e8d27-maas-api-tls\") pod \"maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w\" (UID: \"19995406-ce1f-4562-9de5-5566315e8d27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" Oct 30 12:13:37.477639 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.477493 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19995406-ce1f-4562-9de5-5566315e8d27-openshift-service-ca\") pod \"maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w\" (UID: \"19995406-ce1f-4562-9de5-5566315e8d27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" Oct 30 12:13:37.477639 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.477600 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19995406-ce1f-4562-9de5-5566315e8d27-metrics-certs\") pod \"maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w\" (UID: \"19995406-ce1f-4562-9de5-5566315e8d27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" Oct 30 12:13:37.477760 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.477637 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjg4r\" (UniqueName: \"kubernetes.io/projected/19995406-ce1f-4562-9de5-5566315e8d27-kube-api-access-wjg4r\") pod \"maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w\" (UID: \"19995406-ce1f-4562-9de5-5566315e8d27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" Oct 30 12:13:37.538882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.538849 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:37.579115 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.579019 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19995406-ce1f-4562-9de5-5566315e8d27-maas-api-tls\") pod \"maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w\" (UID: \"19995406-ce1f-4562-9de5-5566315e8d27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" Oct 30 12:13:37.579115 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.579089 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19995406-ce1f-4562-9de5-5566315e8d27-openshift-service-ca\") pod \"maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w\" (UID: \"19995406-ce1f-4562-9de5-5566315e8d27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" Oct 30 12:13:37.579385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.579123 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19995406-ce1f-4562-9de5-5566315e8d27-metrics-certs\") pod \"maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w\" (UID: \"19995406-ce1f-4562-9de5-5566315e8d27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" Oct 30 12:13:37.579385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.579155 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjg4r\" (UniqueName: \"kubernetes.io/projected/19995406-ce1f-4562-9de5-5566315e8d27-kube-api-access-wjg4r\") pod \"maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w\" (UID: \"19995406-ce1f-4562-9de5-5566315e8d27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" Oct 30 12:13:37.579963 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.579937 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19995406-ce1f-4562-9de5-5566315e8d27-openshift-service-ca\") pod \"maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w\" (UID: \"19995406-ce1f-4562-9de5-5566315e8d27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" Oct 30 12:13:37.581847 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.581825 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19995406-ce1f-4562-9de5-5566315e8d27-metrics-certs\") pod \"maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w\" (UID: \"19995406-ce1f-4562-9de5-5566315e8d27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" Oct 30 12:13:37.582001 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.581978 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19995406-ce1f-4562-9de5-5566315e8d27-maas-api-tls\") pod \"maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w\" (UID: \"19995406-ce1f-4562-9de5-5566315e8d27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" Oct 30 12:13:37.615889 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.615856 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjg4r\" (UniqueName: \"kubernetes.io/projected/19995406-ce1f-4562-9de5-5566315e8d27-kube-api-access-wjg4r\") pod \"maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w\" (UID: \"19995406-ce1f-4562-9de5-5566315e8d27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" Oct 30 12:13:37.632840 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.632812 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" Oct 30 12:13:37.939994 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:37.939972 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w"] Oct 30 12:13:37.942563 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:13:37.942534 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19995406_ce1f_4562_9de5_5566315e8d27.slice/crio-64aab1c5c973e68fa9e3f5c264aa6ada85c48ea37794a31bbdb5dbb65ef29d9c WatchSource:0}: Error finding container 64aab1c5c973e68fa9e3f5c264aa6ada85c48ea37794a31bbdb5dbb65ef29d9c: Status 404 returned error can't find the container with id 64aab1c5c973e68fa9e3f5c264aa6ada85c48ea37794a31bbdb5dbb65ef29d9c Oct 30 12:13:38.287839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:38.287762 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:13:38.351332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:38.351296 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" event={"ID":"19995406-ce1f-4562-9de5-5566315e8d27","Type":"ContainerStarted","Data":"64aab1c5c973e68fa9e3f5c264aa6ada85c48ea37794a31bbdb5dbb65ef29d9c"} Oct 30 12:13:39.358798 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:39.358755 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" event={"ID":"19995406-ce1f-4562-9de5-5566315e8d27","Type":"ContainerStarted","Data":"2020cfee499cb527cdfaea78ada417c996d7a0e7faff67f99a480917886264ae"} Oct 30 12:13:39.359304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:39.358933 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" Oct 30 12:13:39.394676 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:39.394645 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:39.402412 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:39.402357 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" podStartSLOduration=1.885604066 podStartE2EDuration="2.402337067s" totalImagesPullingTime="516.733001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:39.39926515 +0000 UTC m=+1172.509108024" watchObservedRunningTime="2026-10-30 12:13:39.402337067 +0000 UTC m=+1172.512179948" Oct 30 12:13:41.194540 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:41.194142 4413 patch_prober.go:29] interesting pod/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": read tcp 10.133.0.2:37956->10.133.0.36:15021: read: connection reset by peer" start-of-body= Oct 30 12:13:41.194540 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:41.194213 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" podUID="4bc64fed-721b-4b8d-91bd-4ca8331409c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": read tcp 10.133.0.2:37956->10.133.0.36:15021: read: connection reset by peer" Oct 30 12:13:41.277587 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:41.277554 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:41.335975 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:41.335951 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" Oct 30 12:13:41.370599 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:41.370541 4413 generic.go:356] "Generic (PLEG): container finished" podID="4bc64fed-721b-4b8d-91bd-4ca8331409c1" containerID="e621ac718a5a1e3696a2553f333583f5dccb2cc4a95403e25264527e09e7e5a6" exitCode=0 Oct 30 12:13:41.370701 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:41.370617 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" event={"ID":"4bc64fed-721b-4b8d-91bd-4ca8331409c1","Type":"ContainerDied","Data":"e621ac718a5a1e3696a2553f333583f5dccb2cc4a95403e25264527e09e7e5a6"} Oct 30 12:13:41.370957 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:41.370944 4413 scope.go:122] "RemoveContainer" containerID="e621ac718a5a1e3696a2553f333583f5dccb2cc4a95403e25264527e09e7e5a6" Oct 30 12:13:41.372217 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:41.372185 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:13:41.372328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:41.372282 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:13:41.372328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:41.372317 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:13:41.403107 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:41.403027 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:41.743541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:41.743442 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:42.377607 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:42.377577 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" event={"ID":"4bc64fed-721b-4b8d-91bd-4ca8331409c1","Type":"ContainerStarted","Data":"a2cd8a8e3a3538b1e772a6d99d59f40c0b72a940600d1a38aa57b7d99686c9e2"} Oct 30 12:13:42.809332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:42.805732 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:42.809332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:42.805868 4413 patch_prober.go:29] interesting pod/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 12:13:42.809332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:42.805916 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" podUID="4bc64fed-721b-4b8d-91bd-4ca8331409c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 12:13:43.462672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:43.462648 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:43.809924 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:43.809825 4413 patch_prober.go:29] interesting pod/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 12:13:43.810691 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:43.810197 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" podUID="4bc64fed-721b-4b8d-91bd-4ca8331409c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 12:13:44.812274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:44.811653 4413 patch_prober.go:29] interesting pod/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 12:13:44.812274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:44.811713 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" podUID="4bc64fed-721b-4b8d-91bd-4ca8331409c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 12:13:45.285547 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:45.285514 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:45.370794 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:45.370770 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" Oct 30 12:13:45.807671 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:45.807631 4413 patch_prober.go:29] interesting pod/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 12:13:45.807967 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:45.807935 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" podUID="4bc64fed-721b-4b8d-91bd-4ca8331409c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 12:13:46.353291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:46.352482 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8" Oct 30 12:13:46.813275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:46.812854 4413 patch_prober.go:29] interesting pod/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 12:13:46.813275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:46.812922 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" podUID="4bc64fed-721b-4b8d-91bd-4ca8331409c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 12:13:47.403532 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:47.403488 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:47.808694 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:47.808616 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:47.808923 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:47.808898 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:47.809605 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:47.809587 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:13:49.088318 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:49.088277 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:51.047581 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:51.047447 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:51.465981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:51.465946 4413 generic.go:356] "Generic (PLEG): container finished" podID="327fe8ba-262d-4b2d-9948-a406ffc77e1c" containerID="34cca965db48c5c7ed3ba167e35a8bb6bdab3852e96f077903689ed06f6cbd58" exitCode=0 Oct 30 12:13:51.466181 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:51.466021 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" event={"ID":"327fe8ba-262d-4b2d-9948-a406ffc77e1c","Type":"ContainerDied","Data":"34cca965db48c5c7ed3ba167e35a8bb6bdab3852e96f077903689ed06f6cbd58"} Oct 30 12:13:51.466492 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:51.466475 4413 scope.go:122] "RemoveContainer" containerID="34cca965db48c5c7ed3ba167e35a8bb6bdab3852e96f077903689ed06f6cbd58" Oct 30 12:13:51.467738 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:51.467710 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:13:51.467812 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:51.467779 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:13:51.467849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:51.467810 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:13:51.895121 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:51.895041 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:52.025168 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:52.025090 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:52.473264 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:52.473217 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" event={"ID":"327fe8ba-262d-4b2d-9948-a406ffc77e1c","Type":"ContainerStarted","Data":"c60e9fa5e1f4e69f32aa975ab30d6e0678f96cf38d4461d704d840ab99d9f2b7"} Oct 30 12:13:53.024434 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:53.024400 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:53.065772 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:53.065739 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:53.067473 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:53.067445 4413 patch_prober.go:29] interesting pod/e2e-ait-e636da41-openshift-default-99df89b57-62xrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 12:13:53.067636 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:53.067502 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" podUID="327fe8ba-262d-4b2d-9948-a406ffc77e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 12:13:54.066417 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:54.066385 4413 patch_prober.go:29] interesting pod/e2e-ait-e636da41-openshift-default-99df89b57-62xrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 12:13:54.066805 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:54.066435 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" podUID="327fe8ba-262d-4b2d-9948-a406ffc77e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 12:13:55.066114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:55.066079 4413 patch_prober.go:29] interesting pod/e2e-ait-e636da41-openshift-default-99df89b57-62xrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 12:13:55.066328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:55.066132 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" podUID="327fe8ba-262d-4b2d-9948-a406ffc77e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 12:13:55.176296 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:55.176239 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:55.825227 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:55.825188 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:56.066462 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:56.066429 4413 patch_prober.go:29] interesting pod/e2e-ait-e636da41-openshift-default-99df89b57-62xrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 12:13:56.066658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:56.066486 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" podUID="327fe8ba-262d-4b2d-9948-a406ffc77e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 12:13:56.230076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:56.230036 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:57.066815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:57.066779 4413 patch_prober.go:29] interesting pod/e2e-ait-e636da41-openshift-default-99df89b57-62xrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 12:13:57.066989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:57.066837 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" podUID="327fe8ba-262d-4b2d-9948-a406ffc77e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 12:13:57.602840 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:57.602806 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:58.070782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:58.070752 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:58.071789 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:58.071766 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:13:58.835651 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:58.835620 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:59.281848 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.281817 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg"] Oct 30 12:13:59.282063 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.282035 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" podUID="b3f923a9-c8d9-40ad-8786-c6058f639fb0" containerName="maas-api" containerID="cri-o://9ccf40639de88ad19f51f641d2b3337e73d508bf00e005388b073ee92e024a25" gracePeriod=30 Oct 30 12:13:59.475012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.474972 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:13:59.535427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.535359 4413 generic.go:356] "Generic (PLEG): container finished" podID="b3f923a9-c8d9-40ad-8786-c6058f639fb0" containerID="9ccf40639de88ad19f51f641d2b3337e73d508bf00e005388b073ee92e024a25" exitCode=0 Oct 30 12:13:59.535528 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.535431 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" event={"ID":"b3f923a9-c8d9-40ad-8786-c6058f639fb0","Type":"ContainerDied","Data":"9ccf40639de88ad19f51f641d2b3337e73d508bf00e005388b073ee92e024a25"} Oct 30 12:13:59.535528 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.535485 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" event={"ID":"b3f923a9-c8d9-40ad-8786-c6058f639fb0","Type":"ContainerDied","Data":"ad74de5fe54f4c0bd601d72aa7e161ce38ef8ea5fa5514ba393b869a215915e2"} Oct 30 12:13:59.535528 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.535502 4413 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad74de5fe54f4c0bd601d72aa7e161ce38ef8ea5fa5514ba393b869a215915e2" Oct 30 12:13:59.541483 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.541465 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" Oct 30 12:13:59.601014 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.600988 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3f923a9-c8d9-40ad-8786-c6058f639fb0-kube-api-access-frpwh\" (UniqueName: \"kubernetes.io/projected/b3f923a9-c8d9-40ad-8786-c6058f639fb0-kube-api-access-frpwh\") pod \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\" (UID: \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\") " Oct 30 12:13:59.601199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.601040 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3f923a9-c8d9-40ad-8786-c6058f639fb0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3f923a9-c8d9-40ad-8786-c6058f639fb0-metrics-certs\") pod \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\" (UID: \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\") " Oct 30 12:13:59.601199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.601117 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3f923a9-c8d9-40ad-8786-c6058f639fb0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3f923a9-c8d9-40ad-8786-c6058f639fb0-maas-api-tls\") pod \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\" (UID: \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\") " Oct 30 12:13:59.601199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.601178 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3f923a9-c8d9-40ad-8786-c6058f639fb0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f923a9-c8d9-40ad-8786-c6058f639fb0-openshift-service-ca\") pod \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\" (UID: \"b3f923a9-c8d9-40ad-8786-c6058f639fb0\") " Oct 30 12:13:59.601609 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.601579 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f923a9-c8d9-40ad-8786-c6058f639fb0-openshift-service-ca" pod "b3f923a9-c8d9-40ad-8786-c6058f639fb0" (UID: "b3f923a9-c8d9-40ad-8786-c6058f639fb0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:13:59.603479 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.603442 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3f923a9-c8d9-40ad-8786-c6058f639fb0-metrics-certs" pod "b3f923a9-c8d9-40ad-8786-c6058f639fb0" (UID: "b3f923a9-c8d9-40ad-8786-c6058f639fb0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:13:59.603479 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.603459 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3f923a9-c8d9-40ad-8786-c6058f639fb0-maas-api-tls" pod "b3f923a9-c8d9-40ad-8786-c6058f639fb0" (UID: "b3f923a9-c8d9-40ad-8786-c6058f639fb0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:13:59.603625 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.603569 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f923a9-c8d9-40ad-8786-c6058f639fb0-kube-api-access-frpwh" pod "b3f923a9-c8d9-40ad-8786-c6058f639fb0" (UID: "b3f923a9-c8d9-40ad-8786-c6058f639fb0"). InnerVolumeSpecName "kube-api-access-frpwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:13:59.702946 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.702904 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3f923a9-c8d9-40ad-8786-c6058f639fb0-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:13:59.703304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.703285 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3f923a9-c8d9-40ad-8786-c6058f639fb0-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:13:59.703489 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.703473 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f923a9-c8d9-40ad-8786-c6058f639fb0-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:13:59.703603 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:13:59.703581 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frpwh\" (UniqueName: \"kubernetes.io/projected/b3f923a9-c8d9-40ad-8786-c6058f639fb0-kube-api-access-frpwh\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:00.071796 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:00.071763 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:00.194174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:00.194139 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:00.321767 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:00.321733 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:00.333342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:00.333278 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5"] Oct 30 12:14:00.333478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:00.333459 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" podUID="4bc64fed-721b-4b8d-91bd-4ca8331409c1" containerName="istio-proxy" containerID="cri-o://a2cd8a8e3a3538b1e772a6d99d59f40c0b72a940600d1a38aa57b7d99686c9e2" gracePeriod=30 Oct 30 12:14:00.454090 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:00.454062 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:00.540405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:00.540375 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg" Oct 30 12:14:00.594213 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:00.594121 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg"] Oct 30 12:14:00.598479 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:00.598450 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-307283d2-7b9d4b9944-jgrkg"] Oct 30 12:14:01.018588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.018554 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3f923a9-c8d9-40ad-8786-c6058f639fb0" path="/var/lib/kubelet/pods/b3f923a9-c8d9-40ad-8786-c6058f639fb0/volumes" Oct 30 12:14:01.223437 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.223398 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:01.342122 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.342044 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:01.666535 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.666503 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm"] Oct 30 12:14:01.667014 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.666993 4413 cpu_manager.go:414] "containerMap: removing container" podUID="b3f923a9-c8d9-40ad-8786-c6058f639fb0" containerName="maas-api" Oct 30 12:14:01.667014 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.667012 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f923a9-c8d9-40ad-8786-c6058f639fb0" containerName="maas-api" Oct 30 12:14:01.667210 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.667114 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f923a9-c8d9-40ad-8786-c6058f639fb0" containerName="maas-api" Oct 30 12:14:01.670754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.670733 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.690647 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.690625 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6a7148-openshift-default-dockercfg-49s7b\"" Oct 30 12:14:01.713888 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.713853 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm"] Oct 30 12:14:01.714063 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.713946 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j"] Oct 30 12:14:01.714158 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.714137 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" podUID="d871951f-2819-4c28-880d-ffc4460b66da" containerName="maas-api" containerID="cri-o://bb32c2c7631bfb35010a8bc0a1f31b08c30e87c8712fbe7f5abc9113f5c7df82" gracePeriod=30 Oct 30 12:14:01.723864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.723828 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.724069 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.724052 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.724141 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.724084 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e867b36-7f5d-4ad3-9192-0149b302b614-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.724141 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.724107 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.724303 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.724230 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.724352 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.724339 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.724386 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.724371 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twc89\" (UniqueName: \"kubernetes.io/projected/7e867b36-7f5d-4ad3-9192-0149b302b614-kube-api-access-twc89\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.724423 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.724405 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.724464 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.724451 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.728910 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.728879 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:01.825926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.825892 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.826098 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.825938 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.826098 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.826068 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.826302 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.826275 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.826426 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.826324 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e867b36-7f5d-4ad3-9192-0149b302b614-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.826426 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.826352 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.826426 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.826351 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.826426 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.826410 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.826660 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.826451 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.826660 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.826480 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twc89\" (UniqueName: \"kubernetes.io/projected/7e867b36-7f5d-4ad3-9192-0149b302b614-kube-api-access-twc89\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.826660 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.826589 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.826841 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.826813 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.826966 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.826941 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.827056 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.827038 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e867b36-7f5d-4ad3-9192-0149b302b614-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.828373 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.828349 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.829201 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.829177 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.850036 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.849996 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:01.850582 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.850548 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twc89\" (UniqueName: \"kubernetes.io/projected/7e867b36-7f5d-4ad3-9192-0149b302b614-kube-api-access-twc89\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.853974 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.853947 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:01.977593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.977567 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:14:01.986765 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:01.986741 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:02.028955 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.028921 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d871951f-2819-4c28-880d-ffc4460b66da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d871951f-2819-4c28-880d-ffc4460b66da-maas-api-tls\") pod \"d871951f-2819-4c28-880d-ffc4460b66da\" (UID: \"d871951f-2819-4c28-880d-ffc4460b66da\") " Oct 30 12:14:02.028955 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.028965 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d871951f-2819-4c28-880d-ffc4460b66da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d871951f-2819-4c28-880d-ffc4460b66da-metrics-certs\") pod \"d871951f-2819-4c28-880d-ffc4460b66da\" (UID: \"d871951f-2819-4c28-880d-ffc4460b66da\") " Oct 30 12:14:02.029199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.029082 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d871951f-2819-4c28-880d-ffc4460b66da-kube-api-access-hjxsg\" (UniqueName: \"kubernetes.io/projected/d871951f-2819-4c28-880d-ffc4460b66da-kube-api-access-hjxsg\") pod \"d871951f-2819-4c28-880d-ffc4460b66da\" (UID: \"d871951f-2819-4c28-880d-ffc4460b66da\") " Oct 30 12:14:02.029199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.029113 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d871951f-2819-4c28-880d-ffc4460b66da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d871951f-2819-4c28-880d-ffc4460b66da-openshift-service-ca\") pod \"d871951f-2819-4c28-880d-ffc4460b66da\" (UID: \"d871951f-2819-4c28-880d-ffc4460b66da\") " Oct 30 12:14:02.029567 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.029535 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d871951f-2819-4c28-880d-ffc4460b66da-openshift-service-ca" pod "d871951f-2819-4c28-880d-ffc4460b66da" (UID: "d871951f-2819-4c28-880d-ffc4460b66da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:02.031546 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.031515 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d871951f-2819-4c28-880d-ffc4460b66da-kube-api-access-hjxsg" pod "d871951f-2819-4c28-880d-ffc4460b66da" (UID: "d871951f-2819-4c28-880d-ffc4460b66da"). InnerVolumeSpecName "kube-api-access-hjxsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:02.033321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.033236 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d871951f-2819-4c28-880d-ffc4460b66da-metrics-certs" pod "d871951f-2819-4c28-880d-ffc4460b66da" (UID: "d871951f-2819-4c28-880d-ffc4460b66da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:14:02.033412 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.033316 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d871951f-2819-4c28-880d-ffc4460b66da-maas-api-tls" pod "d871951f-2819-4c28-880d-ffc4460b66da" (UID: "d871951f-2819-4c28-880d-ffc4460b66da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:14:02.130379 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.130344 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjxsg\" (UniqueName: \"kubernetes.io/projected/d871951f-2819-4c28-880d-ffc4460b66da-kube-api-access-hjxsg\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:02.130379 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.130372 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d871951f-2819-4c28-880d-ffc4460b66da-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:02.130379 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.130382 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d871951f-2819-4c28-880d-ffc4460b66da-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:02.130379 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.130392 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d871951f-2819-4c28-880d-ffc4460b66da-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:02.135488 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.135460 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:02.311507 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.311425 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm"] Oct 30 12:14:02.314705 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:14:02.314677 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e867b36_7f5d_4ad3_9192_0149b302b614.slice/crio-c32f826f02163c3261b43dc7dc47b867b740d8fe35f7a68c9cd53e85ed10b43c WatchSource:0}: Error finding container c32f826f02163c3261b43dc7dc47b867b740d8fe35f7a68c9cd53e85ed10b43c: Status 404 returned error can't find the container with id c32f826f02163c3261b43dc7dc47b867b740d8fe35f7a68c9cd53e85ed10b43c Oct 30 12:14:02.317362 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.317322 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:14:02.317469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.317420 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:14:02.317530 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.317470 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:14:02.532315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.532272 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:02.552817 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.552781 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" event={"ID":"7e867b36-7f5d-4ad3-9192-0149b302b614","Type":"ContainerStarted","Data":"f2f681209ae79ffcbb7e70037a2b2a202e22aa3eb2019c305d28247e0c9e7614"} Oct 30 12:14:02.553011 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.552838 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" event={"ID":"7e867b36-7f5d-4ad3-9192-0149b302b614","Type":"ContainerStarted","Data":"c32f826f02163c3261b43dc7dc47b867b740d8fe35f7a68c9cd53e85ed10b43c"} Oct 30 12:14:02.554340 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.554312 4413 generic.go:356] "Generic (PLEG): container finished" podID="d871951f-2819-4c28-880d-ffc4460b66da" containerID="bb32c2c7631bfb35010a8bc0a1f31b08c30e87c8712fbe7f5abc9113f5c7df82" exitCode=0 Oct 30 12:14:02.554450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.554352 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" Oct 30 12:14:02.554450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.554356 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" event={"ID":"d871951f-2819-4c28-880d-ffc4460b66da","Type":"ContainerDied","Data":"bb32c2c7631bfb35010a8bc0a1f31b08c30e87c8712fbe7f5abc9113f5c7df82"} Oct 30 12:14:02.554450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.554405 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j" event={"ID":"d871951f-2819-4c28-880d-ffc4460b66da","Type":"ContainerDied","Data":"39bf1741eedacc4016ba057193adc0af27d4366f53dde7fcd27ab851c2296c37"} Oct 30 12:14:02.554450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.554437 4413 scope.go:122] "RemoveContainer" containerID="bb32c2c7631bfb35010a8bc0a1f31b08c30e87c8712fbe7f5abc9113f5c7df82" Oct 30 12:14:02.564337 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.564310 4413 scope.go:122] "RemoveContainer" containerID="bb32c2c7631bfb35010a8bc0a1f31b08c30e87c8712fbe7f5abc9113f5c7df82" Oct 30 12:14:02.564737 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:14:02.564716 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb32c2c7631bfb35010a8bc0a1f31b08c30e87c8712fbe7f5abc9113f5c7df82\": container with ID starting with bb32c2c7631bfb35010a8bc0a1f31b08c30e87c8712fbe7f5abc9113f5c7df82 not found: ID does not exist" containerID="bb32c2c7631bfb35010a8bc0a1f31b08c30e87c8712fbe7f5abc9113f5c7df82" Oct 30 12:14:02.564818 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.564746 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb32c2c7631bfb35010a8bc0a1f31b08c30e87c8712fbe7f5abc9113f5c7df82"} err="failed to get container status \"bb32c2c7631bfb35010a8bc0a1f31b08c30e87c8712fbe7f5abc9113f5c7df82\": rpc error: code = NotFound desc = could not find container \"bb32c2c7631bfb35010a8bc0a1f31b08c30e87c8712fbe7f5abc9113f5c7df82\": container with ID starting with bb32c2c7631bfb35010a8bc0a1f31b08c30e87c8712fbe7f5abc9113f5c7df82 not found: ID does not exist" Oct 30 12:14:02.598230 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.598181 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" podStartSLOduration=1.59816848 podStartE2EDuration="1.59816848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:02.597302999 +0000 UTC m=+1195.707145880" watchObservedRunningTime="2026-10-30 12:14:02.59816848 +0000 UTC m=+1195.708011360" Oct 30 12:14:02.633671 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.633623 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j"] Oct 30 12:14:02.638673 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.638639 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c85bd66-f6cf69bf7-nh94j"] Oct 30 12:14:02.675433 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.675397 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:02.795592 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.795562 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:02.811908 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.811865 4413 patch_prober.go:29] interesting pod/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:14:02.812091 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.811925 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" podUID="4bc64fed-721b-4b8d-91bd-4ca8331409c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:14:02.987939 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:02.987911 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:03.020176 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:03.020141 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d871951f-2819-4c28-880d-ffc4460b66da" path="/var/lib/kubelet/pods/d871951f-2819-4c28-880d-ffc4460b66da/volumes" Oct 30 12:14:03.694178 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:03.694147 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8"] Oct 30 12:14:03.816598 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:03.813799 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:03.925100 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:03.925055 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:03.992856 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:03.992782 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:04.416621 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:04.416582 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:04.566069 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:04.566037 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:04.566984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:04.566968 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:04.671970 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:04.671883 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:05.087075 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.086994 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:05.209570 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.209513 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:05.325147 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.325114 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:05.574019 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.573995 4413 generic.go:356] "Generic (PLEG): container finished" podID="4bc64fed-721b-4b8d-91bd-4ca8331409c1" containerID="a2cd8a8e3a3538b1e772a6d99d59f40c0b72a940600d1a38aa57b7d99686c9e2" exitCode=0 Oct 30 12:14:05.574141 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.574065 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" event={"ID":"4bc64fed-721b-4b8d-91bd-4ca8331409c1","Type":"ContainerDied","Data":"a2cd8a8e3a3538b1e772a6d99d59f40c0b72a940600d1a38aa57b7d99686c9e2"} Oct 30 12:14:05.574141 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.574119 4413 scope.go:122] "RemoveContainer" containerID="e621ac718a5a1e3696a2553f333583f5dccb2cc4a95403e25264527e09e7e5a6" Oct 30 12:14:05.598501 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.598480 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:14:05.674583 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.674557 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-credential-socket\") pod \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " Oct 30 12:14:05.674756 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.674613 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-workload-socket\") pod \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " Oct 30 12:14:05.674756 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.674649 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc64fed-721b-4b8d-91bd-4ca8331409c1-kube-api-access-8dj8p\" (UniqueName: \"kubernetes.io/projected/4bc64fed-721b-4b8d-91bd-4ca8331409c1-kube-api-access-8dj8p\") pod \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " Oct 30 12:14:05.674756 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.674688 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-workload-certs\") pod \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " Oct 30 12:14:05.674756 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.674732 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-data\") pod \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " Oct 30 12:14:05.674958 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.674785 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-podinfo\") pod \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " Oct 30 12:14:05.674958 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.674861 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istiod-ca-cert\") pod \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " Oct 30 12:14:05.674958 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.674909 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-credential-socket" pod "4bc64fed-721b-4b8d-91bd-4ca8331409c1" (UID: "4bc64fed-721b-4b8d-91bd-4ca8331409c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:05.674958 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.674926 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-workload-socket" pod "4bc64fed-721b-4b8d-91bd-4ca8331409c1" (UID: "4bc64fed-721b-4b8d-91bd-4ca8331409c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:05.674958 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.674922 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-token\" (UniqueName: \"kubernetes.io/projected/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-token\") pod \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " Oct 30 12:14:05.675171 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.674993 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-envoy\") pod \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\" (UID: \"4bc64fed-721b-4b8d-91bd-4ca8331409c1\") " Oct 30 12:14:05.675171 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.675056 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-data" pod "4bc64fed-721b-4b8d-91bd-4ca8331409c1" (UID: "4bc64fed-721b-4b8d-91bd-4ca8331409c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:05.675171 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.675139 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-workload-certs" pod "4bc64fed-721b-4b8d-91bd-4ca8331409c1" (UID: "4bc64fed-721b-4b8d-91bd-4ca8331409c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:05.675611 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.675532 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:05.675611 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.675555 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:05.675611 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.675573 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:05.675611 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.675587 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:05.675910 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.675622 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istiod-ca-cert" pod "4bc64fed-721b-4b8d-91bd-4ca8331409c1" (UID: "4bc64fed-721b-4b8d-91bd-4ca8331409c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:05.686921 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.677697 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc64fed-721b-4b8d-91bd-4ca8331409c1-kube-api-access-8dj8p" pod "4bc64fed-721b-4b8d-91bd-4ca8331409c1" (UID: "4bc64fed-721b-4b8d-91bd-4ca8331409c1"). InnerVolumeSpecName "kube-api-access-8dj8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:05.686921 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.677905 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-podinfo" pod "4bc64fed-721b-4b8d-91bd-4ca8331409c1" (UID: "4bc64fed-721b-4b8d-91bd-4ca8331409c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:14:05.686921 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.683373 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-envoy" pod "4bc64fed-721b-4b8d-91bd-4ca8331409c1" (UID: "4bc64fed-721b-4b8d-91bd-4ca8331409c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:05.689390 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.689201 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-token" pod "4bc64fed-721b-4b8d-91bd-4ca8331409c1" (UID: "4bc64fed-721b-4b8d-91bd-4ca8331409c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:05.723836 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.723790 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg"] Oct 30 12:14:05.724578 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.724559 4413 cpu_manager.go:414] "containerMap: removing container" podUID="4bc64fed-721b-4b8d-91bd-4ca8331409c1" containerName="istio-proxy" Oct 30 12:14:05.724634 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.724583 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc64fed-721b-4b8d-91bd-4ca8331409c1" containerName="istio-proxy" Oct 30 12:14:05.724634 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.724595 4413 cpu_manager.go:414] "containerMap: removing container" podUID="d871951f-2819-4c28-880d-ffc4460b66da" containerName="maas-api" Oct 30 12:14:05.724634 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.724603 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d871951f-2819-4c28-880d-ffc4460b66da" containerName="maas-api" Oct 30 12:14:05.724634 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.724631 4413 cpu_manager.go:414] "containerMap: removing container" podUID="4bc64fed-721b-4b8d-91bd-4ca8331409c1" containerName="istio-proxy" Oct 30 12:14:05.724752 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.724639 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc64fed-721b-4b8d-91bd-4ca8331409c1" containerName="istio-proxy" Oct 30 12:14:05.724782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.724764 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="d871951f-2819-4c28-880d-ffc4460b66da" containerName="maas-api" Oct 30 12:14:05.724782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.724775 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc64fed-721b-4b8d-91bd-4ca8331409c1" containerName="istio-proxy" Oct 30 12:14:05.724842 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.724789 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc64fed-721b-4b8d-91bd-4ca8331409c1" containerName="istio-proxy" Oct 30 12:14:05.729808 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.729782 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.746701 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.746669 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-512ec5-openshift-default-dockercfg-ww4j8\"" Oct 30 12:14:05.749021 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.748983 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg"] Oct 30 12:14:05.776459 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.776419 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.776459 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.776466 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.776722 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.776491 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc40e968-012f-4ca0-91c1-a67575f59dcd-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.776722 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.776631 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.776839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.776723 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg9t4\" (UniqueName: \"kubernetes.io/projected/bc40e968-012f-4ca0-91c1-a67575f59dcd-kube-api-access-hg9t4\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.776839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.776786 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.776839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.776835 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.777001 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.776919 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.777001 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.776963 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.777113 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.777064 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dj8p\" (UniqueName: \"kubernetes.io/projected/4bc64fed-721b-4b8d-91bd-4ca8331409c1-kube-api-access-8dj8p\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:05.777113 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.777100 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:05.777213 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.777116 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:05.777213 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.777129 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:05.777213 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.777142 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bc64fed-721b-4b8d-91bd-4ca8331409c1-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:05.780091 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.780044 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:05.814635 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:14:05.814595 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:14:05.878174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.878137 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.878374 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.878185 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.878374 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.878231 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.878374 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.878310 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.878374 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.878365 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.878614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.878382 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.878614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.878401 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc40e968-012f-4ca0-91c1-a67575f59dcd-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.878614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.878427 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.878614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.878487 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg9t4\" (UniqueName: \"kubernetes.io/projected/bc40e968-012f-4ca0-91c1-a67575f59dcd-kube-api-access-hg9t4\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.878823 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.878633 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.878883 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.878848 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.879567 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.879055 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.879567 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.879534 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc40e968-012f-4ca0-91c1-a67575f59dcd-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.880211 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.880185 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.881702 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.881683 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.882107 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.882092 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.901348 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.901322 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.901348 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.901339 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg9t4\" (UniqueName: \"kubernetes.io/projected/bc40e968-012f-4ca0-91c1-a67575f59dcd-kube-api-access-hg9t4\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:05.967591 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:05.967556 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:06.044707 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.044673 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:06.357921 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.357894 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg"] Oct 30 12:14:06.359915 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:14:06.359888 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc40e968_012f_4ca0_91c1_a67575f59dcd.slice/crio-21d97e4e5dd8a5eefab9bb51823795c6620676d843f4072e9a0c2dac8a313b91 WatchSource:0}: Error finding container 21d97e4e5dd8a5eefab9bb51823795c6620676d843f4072e9a0c2dac8a313b91: Status 404 returned error can't find the container with id 21d97e4e5dd8a5eefab9bb51823795c6620676d843f4072e9a0c2dac8a313b91 Oct 30 12:14:06.362529 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.362495 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:14:06.362624 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.362564 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:14:06.362624 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.362601 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:14:06.374175 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.374148 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s"] Oct 30 12:14:06.374468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.374441 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" podUID="f0bf1f48-c28e-42d1-9802-9a73475c9946" containerName="maas-api" containerID="cri-o://7d6098ba8d43c1ead6a2d69b08f8b59427adccbc5335b50046c466842d7e2677" gracePeriod=30 Oct 30 12:14:06.449144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.449112 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:06.625527 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.625496 4413 generic.go:356] "Generic (PLEG): container finished" podID="f0bf1f48-c28e-42d1-9802-9a73475c9946" containerID="7d6098ba8d43c1ead6a2d69b08f8b59427adccbc5335b50046c466842d7e2677" exitCode=0 Oct 30 12:14:06.625837 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.625815 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" event={"ID":"f0bf1f48-c28e-42d1-9802-9a73475c9946","Type":"ContainerDied","Data":"7d6098ba8d43c1ead6a2d69b08f8b59427adccbc5335b50046c466842d7e2677"} Oct 30 12:14:06.628143 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.628119 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" event={"ID":"bc40e968-012f-4ca0-91c1-a67575f59dcd","Type":"ContainerStarted","Data":"afd4b9aec95564159b5c65cb2951672a68c9f6cdaa58bf5f7ef9c02724aa7d42"} Oct 30 12:14:06.628304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.628170 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" event={"ID":"bc40e968-012f-4ca0-91c1-a67575f59dcd","Type":"ContainerStarted","Data":"21d97e4e5dd8a5eefab9bb51823795c6620676d843f4072e9a0c2dac8a313b91"} Oct 30 12:14:06.632316 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.631030 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" event={"ID":"4bc64fed-721b-4b8d-91bd-4ca8331409c1","Type":"ContainerDied","Data":"e3e33fffaf57c2da2bcb61865fadf947f782bcfe9d2e78621b365faa5fd73783"} Oct 30 12:14:06.632316 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.631120 4413 scope.go:122] "RemoveContainer" containerID="a2cd8a8e3a3538b1e772a6d99d59f40c0b72a940600d1a38aa57b7d99686c9e2" Oct 30 12:14:06.632316 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.631195 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5" Oct 30 12:14:06.674580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.674535 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" podStartSLOduration=1.674521733 podStartE2EDuration="1.674521733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:06.672676938 +0000 UTC m=+1199.782519818" watchObservedRunningTime="2026-10-30 12:14:06.674521733 +0000 UTC m=+1199.784364677" Oct 30 12:14:06.680805 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.680782 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" Oct 30 12:14:06.739022 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.738832 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5"] Oct 30 12:14:06.742101 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.742070 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-307283d2-gw-openshift-default-5fb6c9f6b5-cccr5"] Oct 30 12:14:06.843514 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.843217 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w"] Oct 30 12:14:06.843987 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.843950 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" podUID="19995406-ce1f-4562-9de5-5566315e8d27" containerName="maas-api" containerID="cri-o://2020cfee499cb527cdfaea78ada417c996d7a0e7faff67f99a480917886264ae" gracePeriod=30 Oct 30 12:14:06.847117 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.845809 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0bf1f48-c28e-42d1-9802-9a73475c9946-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bf1f48-c28e-42d1-9802-9a73475c9946-openshift-service-ca\") pod \"f0bf1f48-c28e-42d1-9802-9a73475c9946\" (UID: \"f0bf1f48-c28e-42d1-9802-9a73475c9946\") " Oct 30 12:14:06.847117 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.845894 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0bf1f48-c28e-42d1-9802-9a73475c9946-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0bf1f48-c28e-42d1-9802-9a73475c9946-maas-api-tls\") pod \"f0bf1f48-c28e-42d1-9802-9a73475c9946\" (UID: \"f0bf1f48-c28e-42d1-9802-9a73475c9946\") " Oct 30 12:14:06.847117 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.845957 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0bf1f48-c28e-42d1-9802-9a73475c9946-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0bf1f48-c28e-42d1-9802-9a73475c9946-metrics-certs\") pod \"f0bf1f48-c28e-42d1-9802-9a73475c9946\" (UID: \"f0bf1f48-c28e-42d1-9802-9a73475c9946\") " Oct 30 12:14:06.847117 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.846021 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0bf1f48-c28e-42d1-9802-9a73475c9946-kube-api-access-42lkg\" (UniqueName: \"kubernetes.io/projected/f0bf1f48-c28e-42d1-9802-9a73475c9946-kube-api-access-42lkg\") pod \"f0bf1f48-c28e-42d1-9802-9a73475c9946\" (UID: \"f0bf1f48-c28e-42d1-9802-9a73475c9946\") " Oct 30 12:14:06.859315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.849844 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bf1f48-c28e-42d1-9802-9a73475c9946-kube-api-access-42lkg" pod "f0bf1f48-c28e-42d1-9802-9a73475c9946" (UID: "f0bf1f48-c28e-42d1-9802-9a73475c9946"). InnerVolumeSpecName "kube-api-access-42lkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:06.859315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.850654 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bf1f48-c28e-42d1-9802-9a73475c9946-openshift-service-ca" pod "f0bf1f48-c28e-42d1-9802-9a73475c9946" (UID: "f0bf1f48-c28e-42d1-9802-9a73475c9946"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:06.859315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.852562 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0bf1f48-c28e-42d1-9802-9a73475c9946-maas-api-tls" pod "f0bf1f48-c28e-42d1-9802-9a73475c9946" (UID: "f0bf1f48-c28e-42d1-9802-9a73475c9946"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:14:06.859315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.854551 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0bf1f48-c28e-42d1-9802-9a73475c9946-metrics-certs" pod "f0bf1f48-c28e-42d1-9802-9a73475c9946" (UID: "f0bf1f48-c28e-42d1-9802-9a73475c9946"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:14:06.870630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.870599 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:06.947456 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.947378 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0bf1f48-c28e-42d1-9802-9a73475c9946-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:06.947456 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.947407 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0bf1f48-c28e-42d1-9802-9a73475c9946-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:06.947456 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.947418 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0bf1f48-c28e-42d1-9802-9a73475c9946-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:06.947456 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:06.947426 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42lkg\" (UniqueName: \"kubernetes.io/projected/f0bf1f48-c28e-42d1-9802-9a73475c9946-kube-api-access-42lkg\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:07.022838 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.022808 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc64fed-721b-4b8d-91bd-4ca8331409c1" path="/var/lib/kubelet/pods/4bc64fed-721b-4b8d-91bd-4ca8331409c1/volumes" Oct 30 12:14:07.045315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.045286 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:07.051819 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.051790 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:07.085484 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.085455 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4"] Oct 30 12:14:07.086184 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.086150 4413 cpu_manager.go:414] "containerMap: removing container" podUID="f0bf1f48-c28e-42d1-9802-9a73475c9946" containerName="maas-api" Oct 30 12:14:07.086184 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.086175 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0bf1f48-c28e-42d1-9802-9a73475c9946" containerName="maas-api" Oct 30 12:14:07.086369 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.086354 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0bf1f48-c28e-42d1-9802-9a73475c9946" containerName="maas-api" Oct 30 12:14:07.086559 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.086539 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:07.091174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.091157 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" Oct 30 12:14:07.092884 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.092862 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" Oct 30 12:14:07.105553 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.105533 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-6a7148\"" Oct 30 12:14:07.122283 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.118143 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4"] Oct 30 12:14:07.213581 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.213507 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:07.249698 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.249668 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19995406-ce1f-4562-9de5-5566315e8d27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/19995406-ce1f-4562-9de5-5566315e8d27-metrics-certs\") pod \"19995406-ce1f-4562-9de5-5566315e8d27\" (UID: \"19995406-ce1f-4562-9de5-5566315e8d27\") " Oct 30 12:14:07.249858 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.249795 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19995406-ce1f-4562-9de5-5566315e8d27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19995406-ce1f-4562-9de5-5566315e8d27-openshift-service-ca\") pod \"19995406-ce1f-4562-9de5-5566315e8d27\" (UID: \"19995406-ce1f-4562-9de5-5566315e8d27\") " Oct 30 12:14:07.249858 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.249822 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19995406-ce1f-4562-9de5-5566315e8d27-kube-api-access-wjg4r\" (UniqueName: \"kubernetes.io/projected/19995406-ce1f-4562-9de5-5566315e8d27-kube-api-access-wjg4r\") pod \"19995406-ce1f-4562-9de5-5566315e8d27\" (UID: \"19995406-ce1f-4562-9de5-5566315e8d27\") " Oct 30 12:14:07.250036 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.249872 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19995406-ce1f-4562-9de5-5566315e8d27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19995406-ce1f-4562-9de5-5566315e8d27-maas-api-tls\") pod \"19995406-ce1f-4562-9de5-5566315e8d27\" (UID: \"19995406-ce1f-4562-9de5-5566315e8d27\") " Oct 30 12:14:07.250036 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.250024 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvd88\" (UniqueName: \"kubernetes.io/projected/40dac033-149e-423e-9630-baeb522263d0-kube-api-access-mvd88\") pod \"payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4\" (UID: \"40dac033-149e-423e-9630-baeb522263d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" Oct 30 12:14:07.250226 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.250198 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19995406-ce1f-4562-9de5-5566315e8d27-openshift-service-ca" pod "19995406-ce1f-4562-9de5-5566315e8d27" (UID: "19995406-ce1f-4562-9de5-5566315e8d27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:07.250315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.250235 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40dac033-149e-423e-9630-baeb522263d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4\" (UID: \"40dac033-149e-423e-9630-baeb522263d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" Oct 30 12:14:07.250403 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.250386 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19995406-ce1f-4562-9de5-5566315e8d27-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:07.252002 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.251976 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19995406-ce1f-4562-9de5-5566315e8d27-kube-api-access-wjg4r" pod "19995406-ce1f-4562-9de5-5566315e8d27" (UID: "19995406-ce1f-4562-9de5-5566315e8d27"). InnerVolumeSpecName "kube-api-access-wjg4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:07.252092 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.252055 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19995406-ce1f-4562-9de5-5566315e8d27-maas-api-tls" pod "19995406-ce1f-4562-9de5-5566315e8d27" (UID: "19995406-ce1f-4562-9de5-5566315e8d27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:14:07.252092 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.252074 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19995406-ce1f-4562-9de5-5566315e8d27-metrics-certs" pod "19995406-ce1f-4562-9de5-5566315e8d27" (UID: "19995406-ce1f-4562-9de5-5566315e8d27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:14:07.351443 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.351397 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40dac033-149e-423e-9630-baeb522263d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4\" (UID: \"40dac033-149e-423e-9630-baeb522263d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" Oct 30 12:14:07.351632 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.351518 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvd88\" (UniqueName: \"kubernetes.io/projected/40dac033-149e-423e-9630-baeb522263d0-kube-api-access-mvd88\") pod \"payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4\" (UID: \"40dac033-149e-423e-9630-baeb522263d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" Oct 30 12:14:07.351684 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.351637 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjg4r\" (UniqueName: \"kubernetes.io/projected/19995406-ce1f-4562-9de5-5566315e8d27-kube-api-access-wjg4r\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:07.351684 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.351653 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19995406-ce1f-4562-9de5-5566315e8d27-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:07.351684 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.351669 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19995406-ce1f-4562-9de5-5566315e8d27-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:07.352564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.352483 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40dac033-149e-423e-9630-baeb522263d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4\" (UID: \"40dac033-149e-423e-9630-baeb522263d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" Oct 30 12:14:07.374729 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.374697 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvd88\" (UniqueName: \"kubernetes.io/projected/40dac033-149e-423e-9630-baeb522263d0-kube-api-access-mvd88\") pod \"payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4\" (UID: \"40dac033-149e-423e-9630-baeb522263d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" Oct 30 12:14:07.404482 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.404455 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" Oct 30 12:14:07.414974 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.414944 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:07.637190 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.637159 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" Oct 30 12:14:07.637390 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.637150 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s" event={"ID":"f0bf1f48-c28e-42d1-9802-9a73475c9946","Type":"ContainerDied","Data":"0c16a48437a7776a9f58ed3e7d1856e9c9047dce73565a0b2f84897bafdbd586"} Oct 30 12:14:07.637390 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.637361 4413 scope.go:122] "RemoveContainer" containerID="7d6098ba8d43c1ead6a2d69b08f8b59427adccbc5335b50046c466842d7e2677" Oct 30 12:14:07.638663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.638549 4413 generic.go:356] "Generic (PLEG): container finished" podID="19995406-ce1f-4562-9de5-5566315e8d27" containerID="2020cfee499cb527cdfaea78ada417c996d7a0e7faff67f99a480917886264ae" exitCode=0 Oct 30 12:14:07.638663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.638606 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" Oct 30 12:14:07.638663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.638630 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" event={"ID":"19995406-ce1f-4562-9de5-5566315e8d27","Type":"ContainerDied","Data":"2020cfee499cb527cdfaea78ada417c996d7a0e7faff67f99a480917886264ae"} Oct 30 12:14:07.638851 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.638683 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w" event={"ID":"19995406-ce1f-4562-9de5-5566315e8d27","Type":"ContainerDied","Data":"64aab1c5c973e68fa9e3f5c264aa6ada85c48ea37794a31bbdb5dbb65ef29d9c"} Oct 30 12:14:07.639185 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.639161 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:07.643502 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.643482 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:07.646571 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.646557 4413 scope.go:122] "RemoveContainer" containerID="2020cfee499cb527cdfaea78ada417c996d7a0e7faff67f99a480917886264ae" Oct 30 12:14:07.654959 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.654943 4413 scope.go:122] "RemoveContainer" containerID="2020cfee499cb527cdfaea78ada417c996d7a0e7faff67f99a480917886264ae" Oct 30 12:14:07.655243 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:14:07.655223 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2020cfee499cb527cdfaea78ada417c996d7a0e7faff67f99a480917886264ae\": container with ID starting with 2020cfee499cb527cdfaea78ada417c996d7a0e7faff67f99a480917886264ae not found: ID does not exist" containerID="2020cfee499cb527cdfaea78ada417c996d7a0e7faff67f99a480917886264ae" Oct 30 12:14:07.655393 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.655371 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2020cfee499cb527cdfaea78ada417c996d7a0e7faff67f99a480917886264ae"} err="failed to get container status \"2020cfee499cb527cdfaea78ada417c996d7a0e7faff67f99a480917886264ae\": rpc error: code = NotFound desc = could not find container \"2020cfee499cb527cdfaea78ada417c996d7a0e7faff67f99a480917886264ae\": container with ID starting with 2020cfee499cb527cdfaea78ada417c996d7a0e7faff67f99a480917886264ae not found: ID does not exist" Oct 30 12:14:07.691304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.691279 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w"] Oct 30 12:14:07.693429 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.693412 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e636da41-58f9d4585d-kxg5w"] Oct 30 12:14:07.709313 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.709294 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4"] Oct 30 12:14:07.711063 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:14:07.711040 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40dac033_149e_423e_9630_baeb522263d0.slice/crio-23537721379d6431dc779a8fbbafb328225eb7dced50a80482534d659fb99c62 WatchSource:0}: Error finding container 23537721379d6431dc779a8fbbafb328225eb7dced50a80482534d659fb99c62: Status 404 returned error can't find the container with id 23537721379d6431dc779a8fbbafb328225eb7dced50a80482534d659fb99c62 Oct 30 12:14:07.719177 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.718999 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:07.773107 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.773082 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s"] Oct 30 12:14:07.774955 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.774927 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80143318-77c9644786-g5m2s"] Oct 30 12:14:07.830504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.830476 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:07.948698 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:07.948613 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:08.266370 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.266274 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:08.424503 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.424471 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:08.518816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.518738 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg"] Oct 30 12:14:08.518988 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.518963 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" podUID="327fe8ba-262d-4b2d-9948-a406ffc77e1c" containerName="istio-proxy" containerID="cri-o://c60e9fa5e1f4e69f32aa975ab30d6e0678f96cf38d4461d704d840ab99d9f2b7" gracePeriod=30 Oct 30 12:14:08.539769 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.539746 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:08.646925 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.646888 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" event={"ID":"40dac033-149e-423e-9630-baeb522263d0","Type":"ContainerStarted","Data":"0973a451e4cf823e4e5c558ce28ce829cc16ba036cc0d218ce264943cccf024f"} Oct 30 12:14:08.647127 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.646948 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" event={"ID":"40dac033-149e-423e-9630-baeb522263d0","Type":"ContainerStarted","Data":"23537721379d6431dc779a8fbbafb328225eb7dced50a80482534d659fb99c62"} Oct 30 12:14:08.647127 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.646982 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" Oct 30 12:14:08.677307 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.677280 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:08.695272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.695216 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8" podUID="d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89" containerName="payload-pre-processing" containerID="cri-o://b26cd6747a335703913bba76db60596f515d5a5738f19c235e1de222fc1e4823" gracePeriod=25 Oct 30 12:14:08.722825 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.720797 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" podStartSLOduration=1.7207772000000001 podStartE2EDuration="1.7207772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:08.719131898 +0000 UTC m=+1201.828974779" watchObservedRunningTime="2026-10-30 12:14:08.7207772 +0000 UTC m=+1201.830620084" Oct 30 12:14:08.754273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.754208 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps"] Oct 30 12:14:08.755067 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.755042 4413 cpu_manager.go:414] "containerMap: removing container" podUID="19995406-ce1f-4562-9de5-5566315e8d27" containerName="maas-api" Oct 30 12:14:08.755067 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.755068 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19995406-ce1f-4562-9de5-5566315e8d27" containerName="maas-api" Oct 30 12:14:08.755218 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.755173 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="19995406-ce1f-4562-9de5-5566315e8d27" containerName="maas-api" Oct 30 12:14:08.759176 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.759147 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.774115 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.774034 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-47fd99-openshift-default-dockercfg-lmqzd\"" Oct 30 12:14:08.785849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.785820 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps"] Oct 30 12:14:08.834854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.834823 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:08.868236 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.868202 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dghxf\" (UniqueName: \"kubernetes.io/projected/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-kube-api-access-dghxf\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.868415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.868261 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.868415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.868296 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.868415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.868334 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.868415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.868380 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.868583 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.868434 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.868583 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.868460 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.868671 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.868644 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.868758 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.868740 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.955412 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.955375 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8" Oct 30 12:14:08.971171 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.971130 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.971384 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.971205 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.971384 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.971232 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.971384 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.971327 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.971384 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.971366 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.971632 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.971432 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.971632 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.971592 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dghxf\" (UniqueName: \"kubernetes.io/projected/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-kube-api-access-dghxf\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.971736 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.971628 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.971736 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.971664 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.973972 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.972572 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.973972 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.973103 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.973972 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.973521 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.973972 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.973875 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.974292 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.974153 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.993007 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.992977 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.994390 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.994365 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.998126 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.998097 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:08.998222 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:08.998149 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dghxf\" (UniqueName: \"kubernetes.io/projected/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-kube-api-access-dghxf\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:09.023508 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.023469 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19995406-ce1f-4562-9de5-5566315e8d27" path="/var/lib/kubelet/pods/19995406-ce1f-4562-9de5-5566315e8d27/volumes" Oct 30 12:14:09.023950 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.023932 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0bf1f48-c28e-42d1-9802-9a73475c9946" path="/var/lib/kubelet/pods/f0bf1f48-c28e-42d1-9802-9a73475c9946/volumes" Oct 30 12:14:09.072641 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.072567 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89-kube-api-access-vj2xz\" (UniqueName: \"kubernetes.io/projected/d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89-kube-api-access-vj2xz\") pod \"d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89\" (UID: \"d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89\") " Oct 30 12:14:09.072795 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.072676 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89-plugins-config-volume\") pod \"d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89\" (UID: \"d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89\") " Oct 30 12:14:09.073097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.073077 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89-plugins-config-volume" pod "d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89" (UID: "d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:09.074886 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.074862 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89-kube-api-access-vj2xz" pod "d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89" (UID: "d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89"). InnerVolumeSpecName "kube-api-access-vj2xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:09.089636 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.089613 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:09.108140 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.108112 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:09.174380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.174351 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj2xz\" (UniqueName: \"kubernetes.io/projected/d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89-kube-api-access-vj2xz\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:09.174503 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.174384 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:09.274689 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.274653 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:09.395189 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.395160 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:09.421233 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.421211 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps"] Oct 30 12:14:09.423177 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:14:09.423152 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7cb8bea_e8fb_4ff2_8d0d_2e6cf8a46da7.slice/crio-92b36127afd714b0b052168d18028ad125ca46e13aa194924e96e4b96d7e4c51 WatchSource:0}: Error finding container 92b36127afd714b0b052168d18028ad125ca46e13aa194924e96e4b96d7e4c51: Status 404 returned error can't find the container with id 92b36127afd714b0b052168d18028ad125ca46e13aa194924e96e4b96d7e4c51 Oct 30 12:14:09.425791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.425760 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:14:09.426072 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.425822 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:14:09.426072 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.425852 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:14:09.518914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.518884 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:09.655539 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.655497 4413 generic.go:356] "Generic (PLEG): container finished" podID="d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89" containerID="b26cd6747a335703913bba76db60596f515d5a5738f19c235e1de222fc1e4823" exitCode=0 Oct 30 12:14:09.655539 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.655528 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8" event={"ID":"d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89","Type":"ContainerDied","Data":"b26cd6747a335703913bba76db60596f515d5a5738f19c235e1de222fc1e4823"} Oct 30 12:14:09.655781 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.655561 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8" Oct 30 12:14:09.655781 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.655585 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8" event={"ID":"d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89","Type":"ContainerDied","Data":"0b5f43c3187016a44cfebaca73c3ae16f62bf5c3b7f10f2fe48cf15b97b57237"} Oct 30 12:14:09.655781 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.655612 4413 scope.go:122] "RemoveContainer" containerID="b26cd6747a335703913bba76db60596f515d5a5738f19c235e1de222fc1e4823" Oct 30 12:14:09.657573 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.657539 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" event={"ID":"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7","Type":"ContainerStarted","Data":"765b3b9754ffe9d27b216e47c9aaaa2a4b858d3f07e4d43374c4916f36eb29ab"} Oct 30 12:14:09.657699 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.657589 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" event={"ID":"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7","Type":"ContainerStarted","Data":"92b36127afd714b0b052168d18028ad125ca46e13aa194924e96e4b96d7e4c51"} Oct 30 12:14:09.667151 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.667132 4413 scope.go:122] "RemoveContainer" containerID="b26cd6747a335703913bba76db60596f515d5a5738f19c235e1de222fc1e4823" Oct 30 12:14:09.667439 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:14:09.667418 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b26cd6747a335703913bba76db60596f515d5a5738f19c235e1de222fc1e4823\": container with ID starting with b26cd6747a335703913bba76db60596f515d5a5738f19c235e1de222fc1e4823 not found: ID does not exist" containerID="b26cd6747a335703913bba76db60596f515d5a5738f19c235e1de222fc1e4823" Oct 30 12:14:09.667508 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.667447 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26cd6747a335703913bba76db60596f515d5a5738f19c235e1de222fc1e4823"} err="failed to get container status \"b26cd6747a335703913bba76db60596f515d5a5738f19c235e1de222fc1e4823\": rpc error: code = NotFound desc = could not find container \"b26cd6747a335703913bba76db60596f515d5a5738f19c235e1de222fc1e4823\": container with ID starting with b26cd6747a335703913bba76db60596f515d5a5738f19c235e1de222fc1e4823 not found: ID does not exist" Oct 30 12:14:09.710296 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.710262 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8"] Oct 30 12:14:09.715281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.715225 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e636da41-77c8b8545f-c7wp8"] Oct 30 12:14:09.759926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.759878 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" podStartSLOduration=1.759863759 podStartE2EDuration="1.759863759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:09.75806978 +0000 UTC m=+1202.867912709" watchObservedRunningTime="2026-10-30 12:14:09.759863759 +0000 UTC m=+1202.869706637" Oct 30 12:14:09.935109 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:09.935036 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:10.048518 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.048492 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp"] Oct 30 12:14:10.048927 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.048915 4413 cpu_manager.go:414] "containerMap: removing container" podUID="d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89" containerName="payload-pre-processing" Oct 30 12:14:10.048981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.048929 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89" containerName="payload-pre-processing" Oct 30 12:14:10.049034 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.049026 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89" containerName="payload-pre-processing" Oct 30 12:14:10.053575 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.053560 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 12:14:10.072681 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.072652 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp"] Oct 30 12:14:10.072982 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.072963 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-6a7148-serving-cert\"" Oct 30 12:14:10.086676 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.086648 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74ccd1d2-3901-46f4-b765-2ace6683fad1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp\" (UID: \"74ccd1d2-3901-46f4-b765-2ace6683fad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 12:14:10.086811 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.086697 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm46c\" (UniqueName: \"kubernetes.io/projected/74ccd1d2-3901-46f4-b765-2ace6683fad1-kube-api-access-lm46c\") pod \"maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp\" (UID: \"74ccd1d2-3901-46f4-b765-2ace6683fad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 12:14:10.087274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.087228 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ccd1d2-3901-46f4-b765-2ace6683fad1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp\" (UID: \"74ccd1d2-3901-46f4-b765-2ace6683fad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 12:14:10.087412 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.087389 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ccd1d2-3901-46f4-b765-2ace6683fad1-metrics-certs\") pod \"maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp\" (UID: \"74ccd1d2-3901-46f4-b765-2ace6683fad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 12:14:10.090296 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.090281 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:10.096180 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.096148 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:10.189167 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.189088 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74ccd1d2-3901-46f4-b765-2ace6683fad1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp\" (UID: \"74ccd1d2-3901-46f4-b765-2ace6683fad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 12:14:10.189167 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.189140 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm46c\" (UniqueName: \"kubernetes.io/projected/74ccd1d2-3901-46f4-b765-2ace6683fad1-kube-api-access-lm46c\") pod \"maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp\" (UID: \"74ccd1d2-3901-46f4-b765-2ace6683fad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 12:14:10.189427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.189233 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ccd1d2-3901-46f4-b765-2ace6683fad1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp\" (UID: \"74ccd1d2-3901-46f4-b765-2ace6683fad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 12:14:10.189427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.189312 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ccd1d2-3901-46f4-b765-2ace6683fad1-metrics-certs\") pod \"maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp\" (UID: \"74ccd1d2-3901-46f4-b765-2ace6683fad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 12:14:10.189427 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:14:10.189235 4413 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-serving-cert: secret "maas-api-e2e-worker-w3-6a7148-serving-cert" not found Oct 30 12:14:10.189427 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:14:10.189404 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ccd1d2-3901-46f4-b765-2ace6683fad1-maas-api-tls podName:74ccd1d2-3901-46f4-b765-2ace6683fad1 nodeName:}" failed. No retries permitted until 2026-10-30 12:14:10.689385921 +0000 UTC m=+1203.799228784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/74ccd1d2-3901-46f4-b765-2ace6683fad1-maas-api-tls") pod "maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" (UID: "74ccd1d2-3901-46f4-b765-2ace6683fad1") : secret "maas-api-e2e-worker-w3-6a7148-serving-cert" not found Oct 30 12:14:10.189925 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.189901 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ccd1d2-3901-46f4-b765-2ace6683fad1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp\" (UID: \"74ccd1d2-3901-46f4-b765-2ace6683fad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 12:14:10.191959 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.191940 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ccd1d2-3901-46f4-b765-2ace6683fad1-metrics-certs\") pod \"maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp\" (UID: \"74ccd1d2-3901-46f4-b765-2ace6683fad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 12:14:10.213165 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.213134 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm46c\" (UniqueName: \"kubernetes.io/projected/74ccd1d2-3901-46f4-b765-2ace6683fad1-kube-api-access-lm46c\") pod \"maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp\" (UID: \"74ccd1d2-3901-46f4-b765-2ace6683fad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 12:14:10.245427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.245392 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:10.361553 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.361525 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:10.664448 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.664419 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:10.665363 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.665346 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:10.695108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.695076 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74ccd1d2-3901-46f4-b765-2ace6683fad1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp\" (UID: \"74ccd1d2-3901-46f4-b765-2ace6683fad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 12:14:10.697527 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.697508 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74ccd1d2-3901-46f4-b765-2ace6683fad1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp\" (UID: \"74ccd1d2-3901-46f4-b765-2ace6683fad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 12:14:10.716759 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.716731 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:10.966554 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:10.966472 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 12:14:11.023537 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:11.023507 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89" path="/var/lib/kubelet/pods/d5aef5d2-76c1-4fec-a9c4-5aff1da8ec89/volumes" Oct 30 12:14:11.128573 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:11.128541 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:11.246865 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:11.246800 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:11.267991 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:11.267970 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp"] Oct 30 12:14:11.269938 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:14:11.269911 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ccd1d2_3901_46f4_b765_2ace6683fad1.slice/crio-5145f13405115d087aa0cb053a17af2f8a0ea3a23224e00019b48a49e0d6829d WatchSource:0}: Error finding container 5145f13405115d087aa0cb053a17af2f8a0ea3a23224e00019b48a49e0d6829d: Status 404 returned error can't find the container with id 5145f13405115d087aa0cb053a17af2f8a0ea3a23224e00019b48a49e0d6829d Oct 30 12:14:11.672826 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:11.672792 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" event={"ID":"74ccd1d2-3901-46f4-b765-2ace6683fad1","Type":"ContainerStarted","Data":"5145f13405115d087aa0cb053a17af2f8a0ea3a23224e00019b48a49e0d6829d"} Oct 30 12:14:12.279425 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:12.279394 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:12.423985 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:12.423951 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:12.547356 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:12.547273 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:12.680421 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:12.680385 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" event={"ID":"74ccd1d2-3901-46f4-b765-2ace6683fad1","Type":"ContainerStarted","Data":"6fa19483225f69b667b0a783c62b9968d9c6d384268c9e964aa235189fdb2018"} Oct 30 12:14:12.680882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:12.680530 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 12:14:12.732330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:12.732274 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" podStartSLOduration=2.374900234 podStartE2EDuration="2.732238397s" totalImagesPullingTime="357.338163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:12.729871263 +0000 UTC m=+1205.839714145" watchObservedRunningTime="2026-10-30 12:14:12.732238397 +0000 UTC m=+1205.842081278" Oct 30 12:14:13.074114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.074077 4413 patch_prober.go:29] interesting pod/e2e-ait-e636da41-openshift-default-99df89b57-62xrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:14:13.074293 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.074128 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" podUID="327fe8ba-262d-4b2d-9948-a406ffc77e1c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:14:13.699789 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.699760 4413 generic.go:356] "Generic (PLEG): container finished" podID="327fe8ba-262d-4b2d-9948-a406ffc77e1c" containerID="c60e9fa5e1f4e69f32aa975ab30d6e0678f96cf38d4461d704d840ab99d9f2b7" exitCode=0 Oct 30 12:14:13.700190 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.699828 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" event={"ID":"327fe8ba-262d-4b2d-9948-a406ffc77e1c","Type":"ContainerDied","Data":"c60e9fa5e1f4e69f32aa975ab30d6e0678f96cf38d4461d704d840ab99d9f2b7"} Oct 30 12:14:13.700190 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.699915 4413 scope.go:122] "RemoveContainer" containerID="34cca965db48c5c7ed3ba167e35a8bb6bdab3852e96f077903689ed06f6cbd58" Oct 30 12:14:13.800886 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.800863 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:14:13.831112 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.831080 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-envoy\") pod \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " Oct 30 12:14:13.831316 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.831137 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-podinfo\") pod \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " Oct 30 12:14:13.831316 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.831193 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-token\" (UniqueName: \"kubernetes.io/projected/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-token\") pod \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " Oct 30 12:14:13.831316 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.831229 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istiod-ca-cert\") pod \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " Oct 30 12:14:13.831316 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.831292 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-workload-socket\") pod \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " Oct 30 12:14:13.831530 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.831336 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-data\") pod \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " Oct 30 12:14:13.831530 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.831405 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-workload-certs\") pod \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " Oct 30 12:14:13.831530 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.831455 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327fe8ba-262d-4b2d-9948-a406ffc77e1c-kube-api-access-cjq7j\" (UniqueName: \"kubernetes.io/projected/327fe8ba-262d-4b2d-9948-a406ffc77e1c-kube-api-access-cjq7j\") pod \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " Oct 30 12:14:13.831683 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.831540 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-credential-socket\") pod \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\" (UID: \"327fe8ba-262d-4b2d-9948-a406ffc77e1c\") " Oct 30 12:14:13.831683 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.831584 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-workload-socket" pod "327fe8ba-262d-4b2d-9948-a406ffc77e1c" (UID: "327fe8ba-262d-4b2d-9948-a406ffc77e1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:13.831683 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.831611 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istiod-ca-cert" pod "327fe8ba-262d-4b2d-9948-a406ffc77e1c" (UID: "327fe8ba-262d-4b2d-9948-a406ffc77e1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:14:13.831832 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.831710 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-data" pod "327fe8ba-262d-4b2d-9948-a406ffc77e1c" (UID: "327fe8ba-262d-4b2d-9948-a406ffc77e1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:13.831832 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.831686 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-workload-certs" pod "327fe8ba-262d-4b2d-9948-a406ffc77e1c" (UID: "327fe8ba-262d-4b2d-9948-a406ffc77e1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:13.831958 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.831926 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-credential-socket" pod "327fe8ba-262d-4b2d-9948-a406ffc77e1c" (UID: "327fe8ba-262d-4b2d-9948-a406ffc77e1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:13.832318 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.832295 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.832407 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.832324 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.832407 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.832341 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.832407 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.832354 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.832407 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.832368 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.833388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.833370 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-podinfo" pod "327fe8ba-262d-4b2d-9948-a406ffc77e1c" (UID: "327fe8ba-262d-4b2d-9948-a406ffc77e1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:14:13.833598 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.833585 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-token" pod "327fe8ba-262d-4b2d-9948-a406ffc77e1c" (UID: "327fe8ba-262d-4b2d-9948-a406ffc77e1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:13.834117 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.834098 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327fe8ba-262d-4b2d-9948-a406ffc77e1c-kube-api-access-cjq7j" pod "327fe8ba-262d-4b2d-9948-a406ffc77e1c" (UID: "327fe8ba-262d-4b2d-9948-a406ffc77e1c"). InnerVolumeSpecName "kube-api-access-cjq7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:14:13.834180 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.834109 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-envoy" pod "327fe8ba-262d-4b2d-9948-a406ffc77e1c" (UID: "327fe8ba-262d-4b2d-9948-a406ffc77e1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:14:13.933738 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.933694 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.933738 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.933731 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.933738 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.933743 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/327fe8ba-262d-4b2d-9948-a406ffc77e1c-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:13.933980 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:13.933756 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjq7j\" (UniqueName: \"kubernetes.io/projected/327fe8ba-262d-4b2d-9948-a406ffc77e1c-kube-api-access-cjq7j\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:14:14.058378 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.058287 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:14.129979 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.129930 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598"] Oct 30 12:14:14.130482 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.130464 4413 cpu_manager.go:414] "containerMap: removing container" podUID="327fe8ba-262d-4b2d-9948-a406ffc77e1c" containerName="istio-proxy" Oct 30 12:14:14.130578 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.130483 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="327fe8ba-262d-4b2d-9948-a406ffc77e1c" containerName="istio-proxy" Oct 30 12:14:14.130578 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.130520 4413 cpu_manager.go:414] "containerMap: removing container" podUID="327fe8ba-262d-4b2d-9948-a406ffc77e1c" containerName="istio-proxy" Oct 30 12:14:14.130578 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.130529 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="327fe8ba-262d-4b2d-9948-a406ffc77e1c" containerName="istio-proxy" Oct 30 12:14:14.130732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.130661 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="327fe8ba-262d-4b2d-9948-a406ffc77e1c" containerName="istio-proxy" Oct 30 12:14:14.134194 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.134175 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 12:14:14.153632 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.153596 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-512ec5-serving-cert\"" Oct 30 12:14:14.156120 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.156099 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598"] Oct 30 12:14:14.187681 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.187643 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:14.236564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.236535 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-metrics-certs\") pod \"maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598\" (UID: \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 12:14:14.236564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.236569 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-maas-api-tls\") pod \"maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598\" (UID: \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 12:14:14.236762 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.236597 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598\" (UID: \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 12:14:14.236799 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.236750 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsjn6\" (UniqueName: \"kubernetes.io/projected/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-kube-api-access-lsjn6\") pod \"maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598\" (UID: \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 12:14:14.298140 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.298101 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:14.337608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.337537 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-metrics-certs\") pod \"maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598\" (UID: \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 12:14:14.337608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.337571 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-maas-api-tls\") pod \"maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598\" (UID: \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 12:14:14.337789 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:14:14.337677 4413 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-serving-cert: secret "maas-api-e2e-worker-w4-512ec5-serving-cert" not found Oct 30 12:14:14.337789 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.337707 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598\" (UID: \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 12:14:14.337789 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:14:14.337734 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-maas-api-tls podName:99360f2e-7ffd-4632-9fbb-2289a1ad6b38 nodeName:}" failed. No retries permitted until 2026-10-30 12:14:14.837712748 +0000 UTC m=+1207.947555622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-maas-api-tls") pod "maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" (UID: "99360f2e-7ffd-4632-9fbb-2289a1ad6b38") : secret "maas-api-e2e-worker-w4-512ec5-serving-cert" not found Oct 30 12:14:14.337906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.337802 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsjn6\" (UniqueName: \"kubernetes.io/projected/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-kube-api-access-lsjn6\") pod \"maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598\" (UID: \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 12:14:14.338335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.338241 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598\" (UID: \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 12:14:14.340178 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.340159 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-metrics-certs\") pod \"maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598\" (UID: \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 12:14:14.365287 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.365050 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsjn6\" (UniqueName: \"kubernetes.io/projected/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-kube-api-access-lsjn6\") pod \"maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598\" (UID: \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 12:14:14.415658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.415626 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:14.557410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.557379 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:14.682948 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.682902 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:14.706020 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.705987 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" Oct 30 12:14:14.706458 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.706019 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg" event={"ID":"327fe8ba-262d-4b2d-9948-a406ffc77e1c","Type":"ContainerDied","Data":"f2e604bfee293e226e362702b2a848aaec9d901ac6d766f1bdf59e6760853877"} Oct 30 12:14:14.706458 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.706072 4413 scope.go:122] "RemoveContainer" containerID="c60e9fa5e1f4e69f32aa975ab30d6e0678f96cf38d4461d704d840ab99d9f2b7" Oct 30 12:14:14.756222 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.756193 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg"] Oct 30 12:14:14.767987 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.767953 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e636da41-openshift-default-99df89b57-62xrg"] Oct 30 12:14:14.842328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.842294 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-maas-api-tls\") pod \"maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598\" (UID: \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 12:14:14.844790 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:14.844774 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-maas-api-tls\") pod \"maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598\" (UID: \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 12:14:15.019116 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:15.019033 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="327fe8ba-262d-4b2d-9948-a406ffc77e1c" path="/var/lib/kubelet/pods/327fe8ba-262d-4b2d-9948-a406ffc77e1c/volumes" Oct 30 12:14:15.047339 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:15.047298 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 12:14:15.351963 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:15.351898 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598"] Oct 30 12:14:15.354310 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:14:15.354277 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99360f2e_7ffd_4632_9fbb_2289a1ad6b38.slice/crio-68f64dd35532e7df6eaaedfc6c6a82f3a7f5f07f0398bf793f8564197aff099e WatchSource:0}: Error finding container 68f64dd35532e7df6eaaedfc6c6a82f3a7f5f07f0398bf793f8564197aff099e: Status 404 returned error can't find the container with id 68f64dd35532e7df6eaaedfc6c6a82f3a7f5f07f0398bf793f8564197aff099e Oct 30 12:14:15.712617 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:15.712584 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" event={"ID":"99360f2e-7ffd-4632-9fbb-2289a1ad6b38","Type":"ContainerStarted","Data":"68f64dd35532e7df6eaaedfc6c6a82f3a7f5f07f0398bf793f8564197aff099e"} Oct 30 12:14:15.928627 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:15.928597 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:16.473144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:16.473084 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:16.593867 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:16.593840 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:16.720262 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:16.720213 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:16.721519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:16.721483 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" event={"ID":"99360f2e-7ffd-4632-9fbb-2289a1ad6b38","Type":"ContainerStarted","Data":"e45cede01bf64208ac6d11f9741bcd82a34d09afbbc79f3f6950d462957215d1"} Oct 30 12:14:16.721737 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:16.721708 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 12:14:16.765529 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:16.765427 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" podStartSLOduration=2.3984215669999998 podStartE2EDuration="2.765412668s" totalImagesPullingTime="366.991101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:16.764042148 +0000 UTC m=+1209.873885029" watchObservedRunningTime="2026-10-30 12:14:16.765412668 +0000 UTC m=+1209.875255549" Oct 30 12:14:16.910766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:16.910735 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:17.729570 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:17.729534 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:18.260487 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.260417 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:18.345507 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.345475 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt"] Oct 30 12:14:18.346329 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.346311 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="327fe8ba-262d-4b2d-9948-a406ffc77e1c" containerName="istio-proxy" Oct 30 12:14:18.350200 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.350184 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" Oct 30 12:14:18.367512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.367486 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-47fd99-serving-cert\"" Oct 30 12:14:18.457275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.456713 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-maas-api-tls\") pod \"maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt\" (UID: \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" Oct 30 12:14:18.457275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.456849 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-metrics-certs\") pod \"maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt\" (UID: \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" Oct 30 12:14:18.457275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.456909 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t847b\" (UniqueName: \"kubernetes.io/projected/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-kube-api-access-t847b\") pod \"maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt\" (UID: \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" Oct 30 12:14:18.457275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.456940 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt\" (UID: \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" Oct 30 12:14:18.460370 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.458435 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:18.464458 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.464430 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt"] Oct 30 12:14:18.558311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.558211 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-metrics-certs\") pod \"maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt\" (UID: \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" Oct 30 12:14:18.558463 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.558307 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t847b\" (UniqueName: \"kubernetes.io/projected/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-kube-api-access-t847b\") pod \"maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt\" (UID: \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" Oct 30 12:14:18.558463 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.558333 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt\" (UID: \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" Oct 30 12:14:18.558463 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.558409 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-maas-api-tls\") pod \"maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt\" (UID: \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" Oct 30 12:14:18.559078 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.559048 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt\" (UID: \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" Oct 30 12:14:18.560806 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.560783 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-metrics-certs\") pod \"maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt\" (UID: \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" Oct 30 12:14:18.560977 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.560957 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-maas-api-tls\") pod \"maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt\" (UID: \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" Oct 30 12:14:18.569106 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.569084 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:18.581082 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.581062 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t847b\" (UniqueName: \"kubernetes.io/projected/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-kube-api-access-t847b\") pod \"maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt\" (UID: \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" Oct 30 12:14:18.666073 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.666037 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" Oct 30 12:14:18.686364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.686288 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:18.747272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:18.746294 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 12:14:19.045966 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:19.045877 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt"] Oct 30 12:14:19.047968 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:14:19.047937 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0198e9a4_ced1_4c15_a6d0_9c9ef3ea1c40.slice/crio-48c61fdb0e5ab02cb0315422804e32bf51438787c20b0ca5b990272119af7ad5 WatchSource:0}: Error finding container 48c61fdb0e5ab02cb0315422804e32bf51438787c20b0ca5b990272119af7ad5: Status 404 returned error can't find the container with id 48c61fdb0e5ab02cb0315422804e32bf51438787c20b0ca5b990272119af7ad5 Oct 30 12:14:19.660685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:19.660659 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" Oct 30 12:14:19.758478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:19.758439 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" event={"ID":"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40","Type":"ContainerStarted","Data":"2fc26a6e03594db198f7b17bceee9099e2a388ecdc1cfad61f99b2c45e272a67"} Oct 30 12:14:19.758930 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:19.758498 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" event={"ID":"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40","Type":"ContainerStarted","Data":"48c61fdb0e5ab02cb0315422804e32bf51438787c20b0ca5b990272119af7ad5"} Oct 30 12:14:19.759334 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:19.759313 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" Oct 30 12:14:19.842276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:19.841538 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" podStartSLOduration=1.528722616 podStartE2EDuration="1.841517874s" totalImagesPullingTime="312.795258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:19.806603368 +0000 UTC m=+1212.916446250" watchObservedRunningTime="2026-10-30 12:14:19.841517874 +0000 UTC m=+1212.951360757" Oct 30 12:14:20.263375 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:20.263336 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:22.046314 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:22.046288 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:22.166404 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:22.166377 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:22.348337 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:22.347940 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:22.743345 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:22.743197 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 12:14:23.926203 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:23.925921 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:24.042397 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:24.042363 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:25.882075 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:25.882040 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:26.774369 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:26.774332 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" Oct 30 12:14:27.849903 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:27.849874 4413 generic.go:356] "Generic (PLEG): container finished" podID="7e867b36-7f5d-4ad3-9192-0149b302b614" containerID="f2f681209ae79ffcbb7e70037a2b2a202e22aa3eb2019c305d28247e0c9e7614" exitCode=0 Oct 30 12:14:27.850383 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:27.849943 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" event={"ID":"7e867b36-7f5d-4ad3-9192-0149b302b614","Type":"ContainerDied","Data":"f2f681209ae79ffcbb7e70037a2b2a202e22aa3eb2019c305d28247e0c9e7614"} Oct 30 12:14:27.850383 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:27.850369 4413 scope.go:122] "RemoveContainer" containerID="f2f681209ae79ffcbb7e70037a2b2a202e22aa3eb2019c305d28247e0c9e7614" Oct 30 12:14:27.851888 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:27.851864 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:14:27.851963 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:27.851950 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:14:27.852004 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:27.851987 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:14:27.877101 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:27.877077 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:28.856448 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:28.856357 4413 generic.go:356] "Generic (PLEG): container finished" podID="bc40e968-012f-4ca0-91c1-a67575f59dcd" containerID="afd4b9aec95564159b5c65cb2951672a68c9f6cdaa58bf5f7ef9c02724aa7d42" exitCode=0 Oct 30 12:14:28.856448 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:28.856437 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" event={"ID":"bc40e968-012f-4ca0-91c1-a67575f59dcd","Type":"ContainerDied","Data":"afd4b9aec95564159b5c65cb2951672a68c9f6cdaa58bf5f7ef9c02724aa7d42"} Oct 30 12:14:28.856953 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:28.856821 4413 scope.go:122] "RemoveContainer" containerID="afd4b9aec95564159b5c65cb2951672a68c9f6cdaa58bf5f7ef9c02724aa7d42" Oct 30 12:14:28.858286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:28.858219 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:14:28.858398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:28.858332 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:14:28.858398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:28.858374 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:14:28.858617 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:28.858599 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" event={"ID":"7e867b36-7f5d-4ad3-9192-0149b302b614","Type":"ContainerStarted","Data":"59c454eeddc9facc39caadf16a6bf27d896ac964ab87b34a9ae24716f4c0f8e5"} Oct 30 12:14:28.989131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:28.987843 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:28.989131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:28.988366 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 12:14:28.989131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:28.988409 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" podUID="7e867b36-7f5d-4ad3-9192-0149b302b614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 12:14:29.324276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:29.323303 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:29.838276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:29.837947 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:29.867742 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:29.867705 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" event={"ID":"bc40e968-012f-4ca0-91c1-a67575f59dcd","Type":"ContainerStarted","Data":"75798be977ee300876bb90f56833a1cfbf4f27b13797a13cae6c83a84183b0f8"} Oct 30 12:14:29.962611 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:29.962573 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:29.987023 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:29.986993 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 12:14:29.987271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:29.987230 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" podUID="7e867b36-7f5d-4ad3-9192-0149b302b614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 12:14:30.046119 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:30.045636 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:30.046119 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:30.045806 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 12:14:30.046119 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:30.045852 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" podUID="bc40e968-012f-4ca0-91c1-a67575f59dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 12:14:30.114158 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:30.114049 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:30.342203 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:30.342025 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:30.987214 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:30.987186 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 12:14:30.987977 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:30.987680 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" podUID="7e867b36-7f5d-4ad3-9192-0149b302b614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 12:14:31.047822 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:31.047657 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 12:14:31.047822 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:31.047717 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" podUID="bc40e968-012f-4ca0-91c1-a67575f59dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 12:14:31.126118 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:31.126042 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:31.263215 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:31.263118 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:31.974806 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:31.974760 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:31.987909 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:31.987876 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 12:14:31.988390 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:31.987931 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" podUID="7e867b36-7f5d-4ad3-9192-0149b302b614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 12:14:32.045144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:32.045109 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 12:14:32.045441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:32.045403 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" podUID="bc40e968-012f-4ca0-91c1-a67575f59dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 12:14:32.143841 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:32.143802 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:32.440306 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:32.440163 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:32.564356 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:32.564189 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:32.988180 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:32.987766 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 12:14:32.988180 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:32.987827 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" podUID="7e867b36-7f5d-4ad3-9192-0149b302b614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 12:14:33.050481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:33.047180 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 12:14:33.050481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:33.047234 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" podUID="bc40e968-012f-4ca0-91c1-a67575f59dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 12:14:33.063314 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:33.063288 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:33.185617 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:33.185585 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:33.343894 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:33.343759 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:33.957008 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:33.956954 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:33.988005 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:33.987973 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 12:14:33.988185 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:33.988026 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" podUID="7e867b36-7f5d-4ad3-9192-0149b302b614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 12:14:34.045852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:34.045804 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 12:14:34.046370 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:34.045875 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" podUID="bc40e968-012f-4ca0-91c1-a67575f59dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 12:14:34.566524 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:34.566485 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:34.911496 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:34.911464 4413 generic.go:356] "Generic (PLEG): container finished" podID="a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" containerID="765b3b9754ffe9d27b216e47c9aaaa2a4b858d3f07e4d43374c4916f36eb29ab" exitCode=0 Oct 30 12:14:34.911716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:34.911535 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" event={"ID":"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7","Type":"ContainerDied","Data":"765b3b9754ffe9d27b216e47c9aaaa2a4b858d3f07e4d43374c4916f36eb29ab"} Oct 30 12:14:34.911992 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:34.911974 4413 scope.go:122] "RemoveContainer" containerID="765b3b9754ffe9d27b216e47c9aaaa2a4b858d3f07e4d43374c4916f36eb29ab" Oct 30 12:14:34.913458 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:34.913428 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:14:34.913517 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:34.913506 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:14:34.913560 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:34.913547 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:14:34.998919 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:34.993302 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:34.998919 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:34.994906 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:14:35.191440 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:35.191410 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:35.192049 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:35.191983 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:35.193049 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:35.193030 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:14:35.835827 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:14:35.835794 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:14:35.917956 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:35.917924 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:35.923888 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:35.923860 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" event={"ID":"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7","Type":"ContainerStarted","Data":"e8e895b77f0f84ed9df1eaacf7692c7578f861388ccfec3b0c2124fd3ab02e05"} Oct 30 12:14:36.090368 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:36.090277 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:36.103420 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:36.103377 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 12:14:36.103861 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:36.103833 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" podUID="a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 12:14:37.103275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:37.099652 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 12:14:37.103275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:37.099716 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" podUID="a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 12:14:37.885897 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:37.885868 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:38.107060 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:38.106599 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 12:14:38.107060 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:38.106664 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" podUID="a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 12:14:39.107275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:39.090326 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 12:14:39.107275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:39.090386 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" podUID="a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 12:14:39.307696 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:39.307660 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:39.838792 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:39.838757 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:39.966079 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:39.966043 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:40.099325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:40.090618 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 12:14:40.099325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:40.090678 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" podUID="a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 12:14:40.127284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:40.123777 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:40.665072 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:40.665042 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:41.096212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:41.096136 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:41.097343 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:41.097324 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 12:14:41.936493 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:41.936459 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:42.262378 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.262288 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn"] Oct 30 12:14:42.267445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.267421 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.295260 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.295215 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-6a7148\"/\"kube-root-ca.crt\"" Oct 30 12:14:42.295260 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.295215 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-6a7148\"/\"openshift-service-ca.crt\"" Oct 30 12:14:42.295462 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.295439 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6a7148\"/\"default-dockercfg-d5dmb\"" Oct 30 12:14:42.295622 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.295598 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6a7148\"/\"simulator-w3-6a7148-kserve-self-signed-certs\"" Oct 30 12:14:42.298771 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.298751 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn"] Oct 30 12:14:42.303105 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.303086 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-tls-certs\") pod \"simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.303170 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.303125 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-home\") pod \"simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.303170 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.303161 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-model-cache\") pod \"simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.303363 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.303342 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-tmp-dir\") pod \"simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.303421 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.303387 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-dshm\") pod \"simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.303471 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.303457 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvq6t\" (UniqueName: \"kubernetes.io/projected/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-kube-api-access-cvq6t\") pod \"simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.329431 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.329407 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:42.404571 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.404548 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-tls-certs\") pod \"simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.404688 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.404591 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-home\") pod \"simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.404688 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.404637 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-model-cache\") pod \"simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.404763 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.404691 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-tmp-dir\") pod \"simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.404763 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.404716 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-dshm\") pod \"simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.404763 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.404742 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvq6t\" (UniqueName: \"kubernetes.io/projected/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-kube-api-access-cvq6t\") pod \"simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.405126 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.405095 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-home\") pod \"simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.405126 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.405114 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-model-cache\") pod \"simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.405309 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.405173 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-tmp-dir\") pod \"simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.407189 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.407166 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-dshm\") pod \"simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.407295 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.407173 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-tls-certs\") pod \"simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.427348 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.427320 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvq6t\" (UniqueName: \"kubernetes.io/projected/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-kube-api-access-cvq6t\") pod \"simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.578636 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.578551 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:42.877037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.877010 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn"] Oct 30 12:14:42.880718 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:14:42.880685 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8ea2a69_24a6_4f0b_adb0_06a662f793fa.slice/crio-d9970e926a9fb3b28fec707e06ab3b581b2175cf5d4083a5e20e489734712212 WatchSource:0}: Error finding container d9970e926a9fb3b28fec707e06ab3b581b2175cf5d4083a5e20e489734712212: Status 404 returned error can't find the container with id d9970e926a9fb3b28fec707e06ab3b581b2175cf5d4083a5e20e489734712212 Oct 30 12:14:42.962546 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.962517 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" event={"ID":"c8ea2a69-24a6-4f0b-adb0-06a662f793fa","Type":"ContainerStarted","Data":"4b0294837a8e0a91b10e05e94f9f81ef86a95a506fb7764fbfcd583e5798f892"} Oct 30 12:14:42.962893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:42.962562 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" event={"ID":"c8ea2a69-24a6-4f0b-adb0-06a662f793fa","Type":"ContainerStarted","Data":"d9970e926a9fb3b28fec707e06ab3b581b2175cf5d4083a5e20e489734712212"} Oct 30 12:14:43.004301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:43.004243 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" podStartSLOduration=1.004231109 podStartE2EDuration="1.004231109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:43.002677594 +0000 UTC m=+1236.112520475" watchObservedRunningTime="2026-10-30 12:14:43.004231109 +0000 UTC m=+1236.114073989" Oct 30 12:14:43.579355 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:43.579325 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:43.847355 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:43.847272 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:45.467689 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:45.467652 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:47.340879 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.340846 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:47.409360 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.409330 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp"] Oct 30 12:14:47.412165 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.412144 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.434718 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.434688 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp"] Oct 30 12:14:47.442215 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.442189 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"openshift-service-ca.crt\"" Oct 30 12:14:47.442440 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.442425 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"kube-root-ca.crt\"" Oct 30 12:14:47.442976 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.442958 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"default-dockercfg-tkn4k\"" Oct 30 12:14:47.443086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.443039 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-47fd99\"/\"simulator-w2-47fd99-kserve-self-signed-certs\"" Oct 30 12:14:47.456199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.456170 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slchw\" (UniqueName: \"kubernetes.io/projected/36c57b1c-6aec-48f0-81aa-adac4196039e-kube-api-access-slchw\") pod \"simulator-w2-47fd99-kserve-55db6bdc4-dtzgp\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.456311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.456230 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36c57b1c-6aec-48f0-81aa-adac4196039e-tls-certs\") pod \"simulator-w2-47fd99-kserve-55db6bdc4-dtzgp\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.456389 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.456311 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-tmp-dir\") pod \"simulator-w2-47fd99-kserve-55db6bdc4-dtzgp\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.456460 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.456441 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-model-cache\") pod \"simulator-w2-47fd99-kserve-55db6bdc4-dtzgp\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.456525 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.456475 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-home\") pod \"simulator-w2-47fd99-kserve-55db6bdc4-dtzgp\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.456581 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.456552 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-dshm\") pod \"simulator-w2-47fd99-kserve-55db6bdc4-dtzgp\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.468543 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.468515 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:47.558154 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.558124 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-home\") pod \"simulator-w2-47fd99-kserve-55db6bdc4-dtzgp\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.558297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.558184 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-dshm\") pod \"simulator-w2-47fd99-kserve-55db6bdc4-dtzgp\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.558297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.558273 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slchw\" (UniqueName: \"kubernetes.io/projected/36c57b1c-6aec-48f0-81aa-adac4196039e-kube-api-access-slchw\") pod \"simulator-w2-47fd99-kserve-55db6bdc4-dtzgp\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.558421 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.558348 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36c57b1c-6aec-48f0-81aa-adac4196039e-tls-certs\") pod \"simulator-w2-47fd99-kserve-55db6bdc4-dtzgp\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.558421 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.558393 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-tmp-dir\") pod \"simulator-w2-47fd99-kserve-55db6bdc4-dtzgp\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.558521 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.558500 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-model-cache\") pod \"simulator-w2-47fd99-kserve-55db6bdc4-dtzgp\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.558637 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.558616 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-home\") pod \"simulator-w2-47fd99-kserve-55db6bdc4-dtzgp\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.558797 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.558782 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-model-cache\") pod \"simulator-w2-47fd99-kserve-55db6bdc4-dtzgp\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.558852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.558779 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-tmp-dir\") pod \"simulator-w2-47fd99-kserve-55db6bdc4-dtzgp\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.560750 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.560727 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-dshm\") pod \"simulator-w2-47fd99-kserve-55db6bdc4-dtzgp\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.560966 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.560950 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36c57b1c-6aec-48f0-81aa-adac4196039e-tls-certs\") pod \"simulator-w2-47fd99-kserve-55db6bdc4-dtzgp\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.580722 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.580701 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slchw\" (UniqueName: \"kubernetes.io/projected/36c57b1c-6aec-48f0-81aa-adac4196039e-kube-api-access-slchw\") pod \"simulator-w2-47fd99-kserve-55db6bdc4-dtzgp\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:47.584612 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.584587 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:47.726698 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:47.726657 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:48.050042 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:48.050019 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp"] Oct 30 12:14:48.051083 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:14:48.051056 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36c57b1c_6aec_48f0_81aa_adac4196039e.slice/crio-6b95687352a450b4a86bde6477bedd53354bd7915728775190b3f42d2d153840 WatchSource:0}: Error finding container 6b95687352a450b4a86bde6477bedd53354bd7915728775190b3f42d2d153840: Status 404 returned error can't find the container with id 6b95687352a450b4a86bde6477bedd53354bd7915728775190b3f42d2d153840 Oct 30 12:14:48.107328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:48.107302 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:48.223992 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:48.223947 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:49.000831 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:49.000799 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" event={"ID":"36c57b1c-6aec-48f0-81aa-adac4196039e","Type":"ContainerStarted","Data":"e82e4708918f6f4a2aaefb9cd588c0eca1a206ee7e54369034a464ade933d301"} Oct 30 12:14:49.001365 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:49.000844 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" event={"ID":"36c57b1c-6aec-48f0-81aa-adac4196039e","Type":"ContainerStarted","Data":"6b95687352a450b4a86bde6477bedd53354bd7915728775190b3f42d2d153840"} Oct 30 12:14:49.041955 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:49.041907 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podStartSLOduration=2.041893235 podStartE2EDuration="2.041893235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:49.041782658 +0000 UTC m=+1242.151625559" watchObservedRunningTime="2026-10-30 12:14:49.041893235 +0000 UTC m=+1242.151736115" Oct 30 12:14:49.362569 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:49.362487 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:49.727206 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:49.727170 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:50.952139 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:50.952088 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:51.104184 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:51.104152 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:52.578985 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:52.578954 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:52.591590 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:52.591567 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:52.969616 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:52.969588 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:53.034031 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:53.034003 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:14:54.665495 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:54.665443 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:55.950823 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:55.950783 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:56.466574 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:56.466542 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:57.727207 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:57.727169 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:57.740115 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:57.740088 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:58.071698 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:58.071624 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:14:58.161967 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:58.161933 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:59.713774 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:59.713724 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:14:59.860984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:14:59.860949 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:01.566698 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.566657 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:01.601198 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.601164 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll"] Oct 30 12:15:01.605683 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.605656 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.624803 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.624767 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll"] Oct 30 12:15:01.632774 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.632739 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"kube-root-ca.crt\"" Oct 30 12:15:01.633018 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.632759 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"openshift-service-ca.crt\"" Oct 30 12:15:01.633181 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.632911 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"default-dockercfg-2vbn8\"" Oct 30 12:15:01.633181 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.633071 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"simulator-w1-c1b1e8-kserve-self-signed-certs\"" Oct 30 12:15:01.690752 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.690714 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:01.717142 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.717103 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-model-cache\") pod \"simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.717388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.717156 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-tls-certs\") pod \"simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.717388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.717363 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-dshm\") pod \"simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.717509 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.717411 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-tmp-dir\") pod \"simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.717509 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.717440 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lf9k\" (UniqueName: \"kubernetes.io/projected/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-kube-api-access-9lf9k\") pod \"simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.717612 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.717517 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-home\") pod \"simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.818786 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.818697 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-home\") pod \"simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.818786 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.818769 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-model-cache\") pod \"simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.818974 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.818809 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-tls-certs\") pod \"simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.818974 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.818909 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-dshm\") pod \"simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.818974 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.818944 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-tmp-dir\") pod \"simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.819082 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.818977 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lf9k\" (UniqueName: \"kubernetes.io/projected/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-kube-api-access-9lf9k\") pod \"simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.819219 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.819198 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-home\") pod \"simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.819304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.819232 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-model-cache\") pod \"simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.819400 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.819372 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-tmp-dir\") pod \"simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.821315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.821296 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-dshm\") pod \"simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.821614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.821595 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-tls-certs\") pod \"simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.845560 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.845523 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lf9k\" (UniqueName: \"kubernetes.io/projected/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-kube-api-access-9lf9k\") pod \"simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:01.920317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:01.920281 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:02.223284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:02.223240 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll"] Oct 30 12:15:02.226055 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:15:02.226030 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85c3372a_f4c6_450f_8de9_5a826d3ee1f6.slice/crio-e96ca24dc6d23e6497f21eda85182f969ae1d4e1496c20e42ec3072a895e8922 WatchSource:0}: Error finding container e96ca24dc6d23e6497f21eda85182f969ae1d4e1496c20e42ec3072a895e8922: Status 404 returned error can't find the container with id e96ca24dc6d23e6497f21eda85182f969ae1d4e1496c20e42ec3072a895e8922 Oct 30 12:15:03.089463 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:03.089430 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" event={"ID":"85c3372a-f4c6-450f-8de9-5a826d3ee1f6","Type":"ContainerStarted","Data":"8759467d1ce0f0147f92eaab43eb2ad310784f198244284ba559cab014dc76be"} Oct 30 12:15:03.089463 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:03.089470 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" event={"ID":"85c3372a-f4c6-450f-8de9-5a826d3ee1f6","Type":"ContainerStarted","Data":"e96ca24dc6d23e6497f21eda85182f969ae1d4e1496c20e42ec3072a895e8922"} Oct 30 12:15:03.130747 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:03.130696 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" podStartSLOduration=2.130680771 podStartE2EDuration="2.130680771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:03.129562858 +0000 UTC m=+1256.239405739" watchObservedRunningTime="2026-10-30 12:15:03.130680771 +0000 UTC m=+1256.240523685" Oct 30 12:15:03.521290 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:03.521203 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:03.921371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:03.921336 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:05.173797 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:05.173758 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:05.814913 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:15:05.814875 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:15:06.855067 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:06.855027 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:07.772570 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:07.772528 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:08.563935 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:08.563897 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:10.288935 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:10.288887 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:11.860478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:11.860441 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:11.920960 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:11.920924 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:11.934430 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:11.934400 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:12.155849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:12.155819 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:15:13.563606 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:13.563556 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:15.276010 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:15.275977 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:17.082749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:17.082683 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:18.769657 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:18.769605 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:18.894869 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:18.894830 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:20.528001 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:20.527971 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:22.252974 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:22.252931 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:23.818422 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:23.818393 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:24.978690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:24.978649 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:25.477103 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:25.477056 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:27.219804 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:27.219765 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:28.942468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:28.942425 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:30.680173 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:30.680134 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:32.348474 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:32.348440 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:34.058173 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:34.058140 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:35.693189 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:35.693151 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:35.815906 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:15:35.815869 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:15:35.888202 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:35.888168 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:37.536947 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:37.536904 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:39.149702 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:39.149654 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:40.174664 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.174630 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft"] Oct 30 12:15:40.177587 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.177567 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.192012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.191994 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f920f7-openshift-default-dockercfg-d89vp\"" Oct 30 12:15:40.206844 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.206812 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft"] Oct 30 12:15:40.252097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.252049 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.252097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.252104 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.252349 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.252186 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.252349 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.252217 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.252349 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.252236 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.252349 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.252312 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.252487 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.252442 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.252487 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.252472 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwbt8\" (UniqueName: \"kubernetes.io/projected/7995a1f6-7cff-4b6a-b612-78b31bad73ba-kube-api-access-cwbt8\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.252565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.252494 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.353292 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.353235 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.353292 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.353295 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwbt8\" (UniqueName: \"kubernetes.io/projected/7995a1f6-7cff-4b6a-b612-78b31bad73ba-kube-api-access-cwbt8\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.353481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.353316 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.353481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.353407 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.353481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.353431 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.353481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.353454 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.353682 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.353484 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.353682 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.353512 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.353682 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.353542 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.353839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.353673 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.353839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.353772 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.353839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.353828 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.353976 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.353958 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.354157 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.354133 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.355905 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.355884 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.356120 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.356102 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.378912 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.378890 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.379001 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.378949 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwbt8\" (UniqueName: \"kubernetes.io/projected/7995a1f6-7cff-4b6a-b612-78b31bad73ba-kube-api-access-cwbt8\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.490316 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.490237 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:40.802516 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.802493 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft"] Oct 30 12:15:40.804295 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:15:40.804259 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7995a1f6_7cff_4b6a_b612_78b31bad73ba.slice/crio-4e3340392dc7cb684fb8fbd4cb0120c33e27772ae2590f719e0c6234b015e095 WatchSource:0}: Error finding container 4e3340392dc7cb684fb8fbd4cb0120c33e27772ae2590f719e0c6234b015e095: Status 404 returned error can't find the container with id 4e3340392dc7cb684fb8fbd4cb0120c33e27772ae2590f719e0c6234b015e095 Oct 30 12:15:40.806666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.806630 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:15:40.806767 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.806727 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:15:40.806826 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.806773 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:15:40.947002 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:40.946964 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:41.317272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:41.316166 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" event={"ID":"7995a1f6-7cff-4b6a-b612-78b31bad73ba","Type":"ContainerStarted","Data":"3a8459d7adeca702e5713789eaa1ed453a6e8cdf7bb8dc58051f3848769399a8"} Oct 30 12:15:41.317272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:41.316235 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" event={"ID":"7995a1f6-7cff-4b6a-b612-78b31bad73ba","Type":"ContainerStarted","Data":"4e3340392dc7cb684fb8fbd4cb0120c33e27772ae2590f719e0c6234b015e095"} Oct 30 12:15:41.387085 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:41.387042 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" podStartSLOduration=1.387028004 podStartE2EDuration="1.387028004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:41.374783958 +0000 UTC m=+1294.484626836" watchObservedRunningTime="2026-10-30 12:15:41.387028004 +0000 UTC m=+1294.496870953" Oct 30 12:15:41.508643 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:41.508605 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:41.514116 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:41.514083 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 12:15:41.514427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:41.514397 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" podUID="7995a1f6-7cff-4b6a-b612-78b31bad73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 12:15:42.515402 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:42.515363 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 12:15:42.515950 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:42.515422 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" podUID="7995a1f6-7cff-4b6a-b612-78b31bad73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 12:15:42.646633 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:42.646596 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:42.816662 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:42.816576 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:43.493761 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:43.493727 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 12:15:43.494076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:43.494040 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" podUID="7995a1f6-7cff-4b6a-b612-78b31bad73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 12:15:44.352653 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:44.352608 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:44.511281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:44.510625 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 12:15:44.511281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:44.510691 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" podUID="7995a1f6-7cff-4b6a-b612-78b31bad73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 12:15:45.511276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:45.507631 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 12:15:45.511276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:45.507693 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" podUID="7995a1f6-7cff-4b6a-b612-78b31bad73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 12:15:45.945571 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:45.945530 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:46.501705 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:46.501668 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 12:15:46.502021 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:46.501994 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" podUID="7995a1f6-7cff-4b6a-b612-78b31bad73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 12:15:47.548994 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:47.548962 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:47.549837 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:47.549815 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:47.550948 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:47.550915 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:15:47.687720 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:47.687677 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:49.462464 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:49.462218 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:51.232279 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:51.232224 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:53.139279 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:53.139215 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:54.922537 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:54.922497 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:56.070965 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:56.070909 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:56.692439 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:56.692408 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:15:58.394794 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:15:58.394734 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:00.049906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:00.049873 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:01.980467 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:01.980431 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:03.750787 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:03.750745 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:05.447454 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:05.447400 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:05.814600 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:16:05.814499 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:16:06.868123 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:06.868074 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:07.049530 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:07.049494 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:08.654605 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:08.654570 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:10.255243 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:10.255203 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:12.042183 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:12.042154 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:13.779703 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:13.779657 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:15.543299 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:15.543236 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:17.300713 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:17.300672 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:17.940166 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:17.940131 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:19.066209 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:19.066178 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:20.767966 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:20.767935 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:22.360036 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:22.360001 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:23.952552 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:23.952517 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:25.691694 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:25.691665 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:27.556651 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:27.556616 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:29.479858 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:29.479778 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:30.804996 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:30.804963 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc"] Oct 30 12:16:30.809080 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:30.809051 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:30.827797 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:30.827770 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc"] Oct 30 12:16:30.837422 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:30.837391 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"openshift-service-ca.crt\"" Oct 30 12:16:30.837422 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:30.837404 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"kube-root-ca.crt\"" Oct 30 12:16:30.837679 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:30.837463 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"e2e-conflict-a-2c88c229-kserve-self-signed-certs\"" Oct 30 12:16:30.837679 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:30.837465 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"default-dockercfg-2qxls\"" Oct 30 12:16:30.856405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:30.856383 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:30.931417 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:30.931386 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-dshm\") pod \"e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:30.931580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:30.931426 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdttq\" (UniqueName: \"kubernetes.io/projected/cf6ed75b-b019-4098-83f5-85211ad2c6fb-kube-api-access-rdttq\") pod \"e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:30.931580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:30.931540 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf6ed75b-b019-4098-83f5-85211ad2c6fb-tls-certs\") pod \"e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:30.931680 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:30.931661 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-model-cache\") pod \"e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:30.931730 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:30.931702 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-tmp-dir\") pod \"e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:30.931766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:30.931760 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-home\") pod \"e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:31.033328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:31.033297 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf6ed75b-b019-4098-83f5-85211ad2c6fb-tls-certs\") pod \"e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:31.033504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:31.033401 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-model-cache\") pod \"e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:31.033504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:31.033447 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-tmp-dir\") pod \"e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:31.033618 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:31.033511 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-home\") pod \"e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:31.033618 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:31.033593 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-dshm\") pod \"e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:31.033734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:31.033628 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdttq\" (UniqueName: \"kubernetes.io/projected/cf6ed75b-b019-4098-83f5-85211ad2c6fb-kube-api-access-rdttq\") pod \"e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:31.033883 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:31.033857 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-model-cache\") pod \"e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:31.033976 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:31.033886 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-tmp-dir\") pod \"e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:31.034027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:31.033987 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-home\") pod \"e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:31.035885 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:31.035860 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-dshm\") pod \"e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:31.036168 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:31.036147 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf6ed75b-b019-4098-83f5-85211ad2c6fb-tls-certs\") pod \"e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:31.056669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:31.056620 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdttq\" (UniqueName: \"kubernetes.io/projected/cf6ed75b-b019-4098-83f5-85211ad2c6fb-kube-api-access-rdttq\") pod \"e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:31.122387 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:31.122364 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:31.345044 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:31.344977 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:31.427657 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:31.427634 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc"] Oct 30 12:16:31.429622 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:16:31.429594 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf6ed75b_b019_4098_83f5_85211ad2c6fb.slice/crio-1f26e88599ec592d7967e497622563aaf5df30f78be03e6e8d53f5d0c8f44636 WatchSource:0}: Error finding container 1f26e88599ec592d7967e497622563aaf5df30f78be03e6e8d53f5d0c8f44636: Status 404 returned error can't find the container with id 1f26e88599ec592d7967e497622563aaf5df30f78be03e6e8d53f5d0c8f44636 Oct 30 12:16:31.588216 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:31.588177 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" event={"ID":"cf6ed75b-b019-4098-83f5-85211ad2c6fb","Type":"ContainerStarted","Data":"afeaa0f8c7875f41f22360e6822222c0c44a1e8f699e0d922c66136113600294"} Oct 30 12:16:31.588216 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:31.588222 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" event={"ID":"cf6ed75b-b019-4098-83f5-85211ad2c6fb","Type":"ContainerStarted","Data":"1f26e88599ec592d7967e497622563aaf5df30f78be03e6e8d53f5d0c8f44636"} Oct 30 12:16:31.630490 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:31.630445 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" podStartSLOduration=1.6304314739999999 podStartE2EDuration="1.630431474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:16:31.628113755 +0000 UTC m=+1344.737956638" watchObservedRunningTime="2026-10-30 12:16:31.630431474 +0000 UTC m=+1344.740274354" Oct 30 12:16:32.123329 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:32.123268 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:33.046415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:33.046386 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:34.781552 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:34.781517 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:35.815972 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:16:35.815930 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:16:36.572731 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:36.572700 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:38.231941 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:38.231907 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:40.074486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:40.074436 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:41.122587 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:41.122558 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:41.135101 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:41.135072 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:41.659496 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:41.659464 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:16:41.747566 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:41.747534 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:43.355670 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:43.355634 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:44.954837 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:44.954803 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:45.803326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:45.803285 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj"] Oct 30 12:16:45.808236 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:45.808210 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:45.825726 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:45.825697 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj"] Oct 30 12:16:45.834619 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:45.834585 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c1b1e8\"/\"distinct-w1-c1b1e8-kserve-self-signed-certs\"" Oct 30 12:16:45.860885 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:45.860849 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:45.915175 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:45.915140 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdebed52-2014-447e-975a-3b6c02c13535-tls-certs\") pod \"distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:45.915355 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:45.915188 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-model-cache\") pod \"distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:45.915355 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:45.915303 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-dshm\") pod \"distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:45.915433 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:45.915378 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-tmp-dir\") pod \"distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:45.915534 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:45.915516 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpctg\" (UniqueName: \"kubernetes.io/projected/cdebed52-2014-447e-975a-3b6c02c13535-kube-api-access-vpctg\") pod \"distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:45.915602 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:45.915580 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-home\") pod \"distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:46.016695 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.016654 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdebed52-2014-447e-975a-3b6c02c13535-tls-certs\") pod \"distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:46.016695 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.016701 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-model-cache\") pod \"distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:46.017270 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.016729 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-dshm\") pod \"distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:46.017270 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.016756 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-tmp-dir\") pod \"distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:46.017270 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.016834 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpctg\" (UniqueName: \"kubernetes.io/projected/cdebed52-2014-447e-975a-3b6c02c13535-kube-api-access-vpctg\") pod \"distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:46.017270 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.016885 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-home\") pod \"distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:46.017488 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.017281 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-model-cache\") pod \"distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:46.017488 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.017332 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-tmp-dir\") pod \"distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:46.017488 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.017348 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-home\") pod \"distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:46.019323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.019301 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-dshm\") pod \"distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:46.019440 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.019425 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdebed52-2014-447e-975a-3b6c02c13535-tls-certs\") pod \"distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:46.040549 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.040516 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpctg\" (UniqueName: \"kubernetes.io/projected/cdebed52-2014-447e-975a-3b6c02c13535-kube-api-access-vpctg\") pod \"distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:46.122832 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.122757 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:46.423412 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.423388 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj"] Oct 30 12:16:46.424992 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:16:46.424966 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdebed52_2014_447e_975a_3b6c02c13535.slice/crio-ff4d87c4affbc7230efc61efa90963d0133e6f6244663ee35973319b0c122cf0 WatchSource:0}: Error finding container ff4d87c4affbc7230efc61efa90963d0133e6f6244663ee35973319b0c122cf0: Status 404 returned error can't find the container with id ff4d87c4affbc7230efc61efa90963d0133e6f6244663ee35973319b0c122cf0 Oct 30 12:16:46.668147 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.668119 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:46.679410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.679339 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" event={"ID":"cdebed52-2014-447e-975a-3b6c02c13535","Type":"ContainerStarted","Data":"7f3652db80dc696daff68775b47bb2f668c9165b6058ba30530bd8a876779441"} Oct 30 12:16:46.679410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.679389 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" event={"ID":"cdebed52-2014-447e-975a-3b6c02c13535","Type":"ContainerStarted","Data":"ff4d87c4affbc7230efc61efa90963d0133e6f6244663ee35973319b0c122cf0"} Oct 30 12:16:46.721900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.721834 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" podStartSLOduration=1.721815074 podStartE2EDuration="1.721815074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:16:46.719861881 +0000 UTC m=+1359.829704762" watchObservedRunningTime="2026-10-30 12:16:46.721815074 +0000 UTC m=+1359.831657956" Oct 30 12:16:46.947684 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.947603 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j"] Oct 30 12:16:46.952420 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.952396 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:46.971445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.971406 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"openshift-service-ca.crt\"" Oct 30 12:16:46.971635 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.971490 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"distinct-w0-f920f7-kserve-self-signed-certs\"" Oct 30 12:16:46.971635 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.971583 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j"] Oct 30 12:16:46.979581 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.979555 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"kube-root-ca.crt\"" Oct 30 12:16:46.979581 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:46.979587 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f920f7\"/\"default-dockercfg-px7j9\"" Oct 30 12:16:47.004593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.004540 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:47.030240 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.030211 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-dshm\") pod \"distinct-w0-f920f7-kserve-6c667ddc4f-qp57j\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.030660 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.030310 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxv8t\" (UniqueName: \"kubernetes.io/projected/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-kube-api-access-lxv8t\") pod \"distinct-w0-f920f7-kserve-6c667ddc4f-qp57j\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.030660 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.030416 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-model-cache\") pod \"distinct-w0-f920f7-kserve-6c667ddc4f-qp57j\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.030660 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.030506 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-home\") pod \"distinct-w0-f920f7-kserve-6c667ddc4f-qp57j\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.030660 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.030574 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-tmp-dir\") pod \"distinct-w0-f920f7-kserve-6c667ddc4f-qp57j\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.030660 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.030637 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-tls-certs\") pod \"distinct-w0-f920f7-kserve-6c667ddc4f-qp57j\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.123950 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.123909 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:47.143275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.131732 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-dshm\") pod \"distinct-w0-f920f7-kserve-6c667ddc4f-qp57j\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.143275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.131824 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxv8t\" (UniqueName: \"kubernetes.io/projected/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-kube-api-access-lxv8t\") pod \"distinct-w0-f920f7-kserve-6c667ddc4f-qp57j\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.143275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.131888 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-model-cache\") pod \"distinct-w0-f920f7-kserve-6c667ddc4f-qp57j\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.143275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.131987 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-home\") pod \"distinct-w0-f920f7-kserve-6c667ddc4f-qp57j\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.143275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.132032 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-tmp-dir\") pod \"distinct-w0-f920f7-kserve-6c667ddc4f-qp57j\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.143275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.132080 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-tls-certs\") pod \"distinct-w0-f920f7-kserve-6c667ddc4f-qp57j\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.143275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.133045 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-model-cache\") pod \"distinct-w0-f920f7-kserve-6c667ddc4f-qp57j\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.143275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.135057 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-home\") pod \"distinct-w0-f920f7-kserve-6c667ddc4f-qp57j\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.143275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.135330 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-tmp-dir\") pod \"distinct-w0-f920f7-kserve-6c667ddc4f-qp57j\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.159325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.154628 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-tls-certs\") pod \"distinct-w0-f920f7-kserve-6c667ddc4f-qp57j\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.159325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.159002 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-dshm\") pod \"distinct-w0-f920f7-kserve-6c667ddc4f-qp57j\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.168558 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.168521 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxv8t\" (UniqueName: \"kubernetes.io/projected/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-kube-api-access-lxv8t\") pod \"distinct-w0-f920f7-kserve-6c667ddc4f-qp57j\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.267145 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.267063 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:47.583342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.583318 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j"] Oct 30 12:16:47.585240 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:16:47.585213 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93cdeb23_4ab6_4708_a0a2_e1bfaac5e3cf.slice/crio-32fe96b509bc2a09d1d343161cfdb851f8252b70a3666220e6871645e1dc398d WatchSource:0}: Error finding container 32fe96b509bc2a09d1d343161cfdb851f8252b70a3666220e6871645e1dc398d: Status 404 returned error can't find the container with id 32fe96b509bc2a09d1d343161cfdb851f8252b70a3666220e6871645e1dc398d Oct 30 12:16:47.686934 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.686898 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" event={"ID":"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf","Type":"ContainerStarted","Data":"71dc899eeb089d4d0e80ae02460b2751bb0bdc30dc9962ab9a172dedeacf256c"} Oct 30 12:16:47.687094 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.686950 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" event={"ID":"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf","Type":"ContainerStarted","Data":"32fe96b509bc2a09d1d343161cfdb851f8252b70a3666220e6871645e1dc398d"} Oct 30 12:16:47.729399 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:47.729333 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podStartSLOduration=1.7293133539999999 podStartE2EDuration="1.729313354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:16:47.727631975 +0000 UTC m=+1360.837474858" watchObservedRunningTime="2026-10-30 12:16:47.729313354 +0000 UTC m=+1360.839156255" Oct 30 12:16:48.267721 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:48.267694 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:48.474289 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:48.474239 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:50.153188 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:50.153150 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:51.755616 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:51.755580 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:53.455371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:53.455332 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:55.478630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:55.478602 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:56.123788 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:56.123761 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:56.136060 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:56.136040 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:56.188616 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:56.188587 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:56.747764 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:56.747736 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:16:57.150399 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:57.150374 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:57.267505 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:57.267478 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:57.280411 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:57.280389 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:57.756160 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:57.756135 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:16:58.303269 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:58.303212 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:16:58.852587 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:16:58.852554 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:00.259669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:00.259633 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:00.648774 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:00.648744 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:02.345159 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:02.345127 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:04.127383 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:04.127321 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:05.754065 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:05.754028 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:05.826980 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:17:05.826944 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:17:07.390532 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:07.390500 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:09.053511 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.053479 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:09.293745 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.293712 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm"] Oct 30 12:17:09.298026 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.298005 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.332697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.332676 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm"] Oct 30 12:17:09.380998 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.380959 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfph9\" (UniqueName: \"kubernetes.io/projected/57785552-cd5b-48b5-b4fc-a62d21e76b8d-kube-api-access-qfph9\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.381142 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.381060 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.381192 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.381159 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.381192 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.381182 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.381312 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.381197 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.381312 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.381220 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.381389 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.381334 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.381427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.381408 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.381538 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.381515 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.482564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.482530 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.482564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.482569 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfph9\" (UniqueName: \"kubernetes.io/projected/57785552-cd5b-48b5-b4fc-a62d21e76b8d-kube-api-access-qfph9\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.482791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.482603 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.482791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.482657 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.482791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.482679 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.482791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.482695 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.482791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.482716 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.483031 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.483005 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.483119 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.483095 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.483182 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.483133 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.483235 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.483202 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.483235 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.483133 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.483397 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.483373 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.483565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.483547 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.485195 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.485179 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.485639 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.485623 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.505320 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.505292 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.505504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.505485 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfph9\" (UniqueName: \"kubernetes.io/projected/57785552-cd5b-48b5-b4fc-a62d21e76b8d-kube-api-access-qfph9\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.612479 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.612383 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:09.933208 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.933181 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm"] Oct 30 12:17:09.934197 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:17:09.934168 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57785552_cd5b_48b5_b4fc_a62d21e76b8d.slice/crio-8694d9ab5150b314f69660e6db2d3f43ebb26642687e8753f7a131d2287d3a86 WatchSource:0}: Error finding container 8694d9ab5150b314f69660e6db2d3f43ebb26642687e8753f7a131d2287d3a86: Status 404 returned error can't find the container with id 8694d9ab5150b314f69660e6db2d3f43ebb26642687e8753f7a131d2287d3a86 Oct 30 12:17:09.936526 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.936496 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:17:09.936595 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.936562 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:17:09.936595 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:09.936590 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:17:10.371580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:10.371497 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:10.712236 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:10.712194 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:10.823945 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:10.823909 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" event={"ID":"57785552-cd5b-48b5-b4fc-a62d21e76b8d","Type":"ContainerStarted","Data":"d32f2cf6cb59a8ea5cafa3fc53bbb34239885f19543c9a70825d859916ecccb5"} Oct 30 12:17:10.824136 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:10.823972 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" event={"ID":"57785552-cd5b-48b5-b4fc-a62d21e76b8d","Type":"ContainerStarted","Data":"8694d9ab5150b314f69660e6db2d3f43ebb26642687e8753f7a131d2287d3a86"} Oct 30 12:17:10.828455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:10.828428 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:10.880591 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:10.880547 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" podStartSLOduration=1.880531812 podStartE2EDuration="1.880531812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:10.878952599 +0000 UTC m=+1383.988795479" watchObservedRunningTime="2026-10-30 12:17:10.880531812 +0000 UTC m=+1383.990374693" Oct 30 12:17:11.615363 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:11.615323 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:11.616049 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:11.616017 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 12:17:11.616202 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:11.616067 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" podUID="57785552-cd5b-48b5-b4fc-a62d21e76b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 12:17:12.445033 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:12.444994 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:12.615275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:12.614662 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 12:17:12.615275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:12.614726 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" podUID="57785552-cd5b-48b5-b4fc-a62d21e76b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 12:17:13.617274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:13.616606 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 12:17:13.617274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:13.616670 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" podUID="57785552-cd5b-48b5-b4fc-a62d21e76b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 12:17:14.050875 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:14.050844 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:14.615628 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:14.615595 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 12:17:14.615806 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:14.615663 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" podUID="57785552-cd5b-48b5-b4fc-a62d21e76b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 12:17:15.615652 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:15.615612 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 12:17:15.616059 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:15.615669 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" podUID="57785552-cd5b-48b5-b4fc-a62d21e76b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 12:17:15.748625 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:15.748578 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:16.617843 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:16.617813 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:16.618230 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:16.618100 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:16.618958 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:16.618943 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:17:17.466510 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:17.466464 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:18.780821 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.780790 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm"] Oct 30 12:17:18.785405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.785385 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.804452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.804422 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c1b1e8-openshift-default-dockercfg-k4lgw\"" Oct 30 12:17:18.812337 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.812312 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.812459 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.812385 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.812459 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.812429 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.812580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.812552 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.812689 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.812666 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.812859 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.812837 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.812927 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.812876 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e35374-76c0-476a-bfcf-0a3e253718ee-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.812982 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.812947 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.813039 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.812988 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf2dj\" (UniqueName: \"kubernetes.io/projected/18e35374-76c0-476a-bfcf-0a3e253718ee-kube-api-access-rf2dj\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.817037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.817011 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm"] Oct 30 12:17:18.915274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.914582 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.923271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.915130 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e35374-76c0-476a-bfcf-0a3e253718ee-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.923271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.915945 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.923271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.915986 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf2dj\" (UniqueName: \"kubernetes.io/projected/18e35374-76c0-476a-bfcf-0a3e253718ee-kube-api-access-rf2dj\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.923271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.916028 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.923271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.916119 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.923271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.916146 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.923271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.916179 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.923271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.916233 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.923271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.915057 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.923271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.916733 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.923271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.915842 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e35374-76c0-476a-bfcf-0a3e253718ee-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.923271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.917233 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.923271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.917518 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.923271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.920562 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.923271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.920777 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.943933 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.943856 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:18.945709 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:18.945653 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf2dj\" (UniqueName: \"kubernetes.io/projected/18e35374-76c0-476a-bfcf-0a3e253718ee-kube-api-access-rf2dj\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:19.103486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:19.102971 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:19.357635 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:19.357539 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:19.482139 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:19.482113 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm"] Oct 30 12:17:19.484160 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:17:19.484137 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e35374_76c0_476a_bfcf_0a3e253718ee.slice/crio-dd0ae584e7d61b669a3dceedc8b696d05ecd3ef92dc68628f5a0d9deb69b4988 WatchSource:0}: Error finding container dd0ae584e7d61b669a3dceedc8b696d05ecd3ef92dc68628f5a0d9deb69b4988: Status 404 returned error can't find the container with id dd0ae584e7d61b669a3dceedc8b696d05ecd3ef92dc68628f5a0d9deb69b4988 Oct 30 12:17:19.489812 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:19.489780 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:17:19.489881 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:19.489847 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:17:19.489881 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:19.489876 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:17:19.885729 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:19.885691 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" event={"ID":"18e35374-76c0-476a-bfcf-0a3e253718ee","Type":"ContainerStarted","Data":"05197d46f7b85982dd412718fc53b721aebc2f87439b4ace83eea8d6fab9a01b"} Oct 30 12:17:19.886133 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:19.885744 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" event={"ID":"18e35374-76c0-476a-bfcf-0a3e253718ee","Type":"ContainerStarted","Data":"dd0ae584e7d61b669a3dceedc8b696d05ecd3ef92dc68628f5a0d9deb69b4988"} Oct 30 12:17:19.932044 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:19.931995 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" podStartSLOduration=1.931981553 podStartE2EDuration="1.931981553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:19.929464283 +0000 UTC m=+1393.039307182" watchObservedRunningTime="2026-10-30 12:17:19.931981553 +0000 UTC m=+1393.041824434" Oct 30 12:17:20.103704 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:20.103667 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:20.105235 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:20.105214 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 12:17:20.105324 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:20.105298 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" podUID="18e35374-76c0-476a-bfcf-0a3e253718ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 12:17:20.619048 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:20.618996 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:21.060859 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:21.060823 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:21.113274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:21.109688 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 12:17:21.113274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:21.109751 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" podUID="18e35374-76c0-476a-bfcf-0a3e253718ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 12:17:22.106273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:22.104567 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 12:17:22.106273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:22.104624 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" podUID="18e35374-76c0-476a-bfcf-0a3e253718ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 12:17:22.756008 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:22.755976 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:23.115438 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:23.115347 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 12:17:23.115438 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:23.115407 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" podUID="18e35374-76c0-476a-bfcf-0a3e253718ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 12:17:24.112278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:24.104692 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 12:17:24.112278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:24.104757 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" podUID="18e35374-76c0-476a-bfcf-0a3e253718ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 12:17:24.454599 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:24.454553 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:25.111751 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:25.111710 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 12:17:25.112078 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:25.112044 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" podUID="18e35374-76c0-476a-bfcf-0a3e253718ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 12:17:26.103912 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:26.103876 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 12:17:26.104342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:26.103929 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" podUID="18e35374-76c0-476a-bfcf-0a3e253718ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 12:17:26.148614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:26.148580 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:27.108066 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:27.108040 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:27.108582 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:27.108432 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:27.109187 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:27.109171 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:17:27.850484 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:27.850443 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:28.214729 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:28.214700 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:28.329507 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:28.329476 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:29.607707 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:29.607677 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:31.257726 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:31.257693 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:32.146062 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:32.146028 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:33.182603 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:33.182573 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:34.611903 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:34.611873 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:34.971313 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:34.971277 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:35.814183 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:17:35.814142 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:17:36.344393 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.344360 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7"] Oct 30 12:17:36.348432 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.348409 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.376764 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.376734 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.376764 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.376766 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/682d04e3-a053-4d89-a812-d7669ed7c595-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.377049 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.376790 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/682d04e3-a053-4d89-a812-d7669ed7c595-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.377049 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.376811 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/682d04e3-a053-4d89-a812-d7669ed7c595-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.377049 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.377015 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.377221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.377066 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.377221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.377158 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl4cp\" (UniqueName: \"kubernetes.io/projected/682d04e3-a053-4d89-a812-d7669ed7c595-kube-api-access-bl4cp\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.377221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.377191 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.377413 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.377362 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.379917 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.379894 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7"] Oct 30 12:17:36.479164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.479130 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.479164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.479161 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/682d04e3-a053-4d89-a812-d7669ed7c595-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.479469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.479180 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/682d04e3-a053-4d89-a812-d7669ed7c595-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.479469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.479201 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/682d04e3-a053-4d89-a812-d7669ed7c595-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.479469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.479223 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.479469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.479244 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.479469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.479308 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl4cp\" (UniqueName: \"kubernetes.io/projected/682d04e3-a053-4d89-a812-d7669ed7c595-kube-api-access-bl4cp\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.479469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.479333 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.479469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.479426 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.479865 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.479637 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.479865 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.479802 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.479865 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.479827 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.479991 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.479932 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.480040 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.480002 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/682d04e3-a053-4d89-a812-d7669ed7c595-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.481788 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.481766 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.482088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.482067 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/682d04e3-a053-4d89-a812-d7669ed7c595-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.508888 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.502736 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/682d04e3-a053-4d89-a812-d7669ed7c595-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.508888 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.502949 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl4cp\" (UniqueName: \"kubernetes.io/projected/682d04e3-a053-4d89-a812-d7669ed7c595-kube-api-access-bl4cp\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.598617 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.598533 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:36.665087 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.665053 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:36.996938 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:36.996915 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7"] Oct 30 12:17:36.998962 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:17:36.998933 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod682d04e3_a053_4d89_a812_d7669ed7c595.slice/crio-ddbabf0b86b0cf1917d60354bcd910b04485b26934643b2f061e2fd5ba22f7fe WatchSource:0}: Error finding container ddbabf0b86b0cf1917d60354bcd910b04485b26934643b2f061e2fd5ba22f7fe: Status 404 returned error can't find the container with id ddbabf0b86b0cf1917d60354bcd910b04485b26934643b2f061e2fd5ba22f7fe Oct 30 12:17:37.002241 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:37.002205 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:17:37.002323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:37.002307 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:17:37.002359 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:37.002340 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:17:37.009474 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:37.009451 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" event={"ID":"682d04e3-a053-4d89-a812-d7669ed7c595","Type":"ContainerStarted","Data":"ddbabf0b86b0cf1917d60354bcd910b04485b26934643b2f061e2fd5ba22f7fe"} Oct 30 12:17:38.026686 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:38.026646 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" event={"ID":"682d04e3-a053-4d89-a812-d7669ed7c595","Type":"ContainerStarted","Data":"943bfc8852004c1793df262241811100d3a4d6c5ae3636cb2dba79f3477778c7"} Oct 30 12:17:38.115277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:38.113375 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" podStartSLOduration=2.11335393 podStartE2EDuration="2.11335393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:38.082129958 +0000 UTC m=+1411.191972839" watchObservedRunningTime="2026-10-30 12:17:38.11335393 +0000 UTC m=+1411.223196812" Oct 30 12:17:38.280766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:38.280685 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:38.665741 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:38.665702 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:38.667522 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:38.667498 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 12:17:38.667676 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:38.667553 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" podUID="682d04e3-a053-4d89-a812-d7669ed7c595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 12:17:39.665895 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:39.665861 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 12:17:39.666337 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:39.665921 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" podUID="682d04e3-a053-4d89-a812-d7669ed7c595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 12:17:40.149468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:40.149439 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:40.666345 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:40.666311 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 12:17:40.666842 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:40.666373 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" podUID="682d04e3-a053-4d89-a812-d7669ed7c595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 12:17:41.666212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:41.666180 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 12:17:41.666446 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:41.666244 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" podUID="682d04e3-a053-4d89-a812-d7669ed7c595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 12:17:42.389940 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:42.389908 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:42.665971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:42.665889 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 12:17:42.665971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:42.665950 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" podUID="682d04e3-a053-4d89-a812-d7669ed7c595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 12:17:43.666138 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:43.666106 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 12:17:43.666552 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:43.666167 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" podUID="682d04e3-a053-4d89-a812-d7669ed7c595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 12:17:44.109751 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:44.109660 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:45.404451 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:45.404419 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:45.404937 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:45.404700 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:45.405546 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:45.405523 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:17:45.901265 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:45.901225 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:47.770837 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:47.770792 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:49.361050 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:49.361016 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:49.782663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:49.782616 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc"] Oct 30 12:17:49.828945 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:49.828912 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:51.063209 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:51.063152 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:52.752075 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:52.752043 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:54.459668 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:54.459640 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:54.849704 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:54.849592 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:55.566441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.566409 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj"] Oct 30 12:17:55.572277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.572240 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.592849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.592824 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj"] Oct 30 12:17:55.634159 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.634129 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78cwb\" (UniqueName: \"kubernetes.io/projected/3a8cf876-a088-4dd2-9890-27772782da71-kube-api-access-78cwb\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.634337 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.634293 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a8cf876-a088-4dd2-9890-27772782da71-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.634337 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.634321 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a8cf876-a088-4dd2-9890-27772782da71-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.634415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.634345 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a8cf876-a088-4dd2-9890-27772782da71-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.634415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.634369 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.634507 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.634484 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.634544 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.634530 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.634664 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.634639 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.634747 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.634718 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.735936 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.735901 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78cwb\" (UniqueName: \"kubernetes.io/projected/3a8cf876-a088-4dd2-9890-27772782da71-kube-api-access-78cwb\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.736115 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.736007 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a8cf876-a088-4dd2-9890-27772782da71-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.736115 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.736029 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a8cf876-a088-4dd2-9890-27772782da71-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.736115 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.736067 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a8cf876-a088-4dd2-9890-27772782da71-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.736115 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.736097 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.736572 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.736231 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.736572 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.736312 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.736572 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.736371 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.736572 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.736436 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.736786 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.736651 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.736786 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.736746 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.736898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.736844 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.736898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.736878 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.737006 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.736920 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a8cf876-a088-4dd2-9890-27772782da71-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.739027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.739003 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a8cf876-a088-4dd2-9890-27772782da71-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.739148 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.739029 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.761611 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.761581 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a8cf876-a088-4dd2-9890-27772782da71-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.761776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.761702 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78cwb\" (UniqueName: \"kubernetes.io/projected/3a8cf876-a088-4dd2-9890-27772782da71-kube-api-access-78cwb\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:55.888540 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:55.888499 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:56.268336 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:56.264813 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj"] Oct 30 12:17:56.275681 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:56.275643 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:17:56.275811 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:56.275718 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:17:56.275811 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:56.275747 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:17:56.345374 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:56.342739 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:56.476281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:56.475156 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:57.145656 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:57.145609 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" event={"ID":"3a8cf876-a088-4dd2-9890-27772782da71","Type":"ContainerStarted","Data":"5bcd85c8fec1de4937e11c880f4f2d41a12d73824ec12fe69d89869c1dc1f593"} Oct 30 12:17:57.146084 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:57.145689 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" event={"ID":"3a8cf876-a088-4dd2-9890-27772782da71","Type":"ContainerStarted","Data":"3bcaa8576049114c4ae96d990119e28cd8eb2af2ae498cc5d72befcda22d333f"} Oct 30 12:17:57.189712 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:57.189664 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" podStartSLOduration=2.189642322 podStartE2EDuration="2.189642322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:57.188992541 +0000 UTC m=+1430.298835423" watchObservedRunningTime="2026-10-30 12:17:57.189642322 +0000 UTC m=+1430.299485203" Oct 30 12:17:57.889210 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:57.889180 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:17:57.915708 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:57.915674 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 12:17:57.915879 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:57.915731 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" podUID="3a8cf876-a088-4dd2-9890-27772782da71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 12:17:58.211332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:58.209243 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:17:58.890226 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:58.890193 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 12:17:58.890468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:58.890282 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" podUID="3a8cf876-a088-4dd2-9890-27772782da71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 12:17:59.889958 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:59.889924 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 12:17:59.890387 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:59.890019 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" podUID="3a8cf876-a088-4dd2-9890-27772782da71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 12:17:59.915290 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:17:59.910178 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:00.889855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:00.889816 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 12:18:00.890013 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:00.889875 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" podUID="3a8cf876-a088-4dd2-9890-27772782da71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 12:18:01.671131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:01.671084 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:01.889366 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:01.889333 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 12:18:01.889553 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:01.889391 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" podUID="3a8cf876-a088-4dd2-9890-27772782da71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 12:18:02.890006 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:02.889974 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 12:18:02.890517 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:02.890029 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" podUID="3a8cf876-a088-4dd2-9890-27772782da71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 12:18:03.348118 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:03.348037 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:03.894236 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:03.894203 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:18:03.894711 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:03.894517 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:18:03.895360 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:03.895344 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:18:04.803887 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:04.803852 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" podUID="cf6ed75b-b019-4098-83f5-85211ad2c6fb" containerName="main" containerID="cri-o://afeaa0f8c7875f41f22360e6822222c0c44a1e8f699e0d922c66136113600294" gracePeriod=45 Oct 30 12:18:04.951206 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:04.951171 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:05.077874 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.077850 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:18:05.192717 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.192683 4413 generic.go:356] "Generic (PLEG): container finished" podID="cf6ed75b-b019-4098-83f5-85211ad2c6fb" containerID="afeaa0f8c7875f41f22360e6822222c0c44a1e8f699e0d922c66136113600294" exitCode=0 Oct 30 12:18:05.192912 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.192735 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" Oct 30 12:18:05.192912 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.192772 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" event={"ID":"cf6ed75b-b019-4098-83f5-85211ad2c6fb","Type":"ContainerDied","Data":"afeaa0f8c7875f41f22360e6822222c0c44a1e8f699e0d922c66136113600294"} Oct 30 12:18:05.192912 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.192827 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc" event={"ID":"cf6ed75b-b019-4098-83f5-85211ad2c6fb","Type":"ContainerDied","Data":"1f26e88599ec592d7967e497622563aaf5df30f78be03e6e8d53f5d0c8f44636"} Oct 30 12:18:05.192912 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.192852 4413 scope.go:122] "RemoveContainer" containerID="afeaa0f8c7875f41f22360e6822222c0c44a1e8f699e0d922c66136113600294" Oct 30 12:18:05.202735 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.202682 4413 scope.go:122] "RemoveContainer" containerID="afeaa0f8c7875f41f22360e6822222c0c44a1e8f699e0d922c66136113600294" Oct 30 12:18:05.203021 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:18:05.203001 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afeaa0f8c7875f41f22360e6822222c0c44a1e8f699e0d922c66136113600294\": container with ID starting with afeaa0f8c7875f41f22360e6822222c0c44a1e8f699e0d922c66136113600294 not found: ID does not exist" containerID="afeaa0f8c7875f41f22360e6822222c0c44a1e8f699e0d922c66136113600294" Oct 30 12:18:05.203095 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.203031 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afeaa0f8c7875f41f22360e6822222c0c44a1e8f699e0d922c66136113600294"} err="failed to get container status \"afeaa0f8c7875f41f22360e6822222c0c44a1e8f699e0d922c66136113600294\": rpc error: code = NotFound desc = could not find container \"afeaa0f8c7875f41f22360e6822222c0c44a1e8f699e0d922c66136113600294\": container with ID starting with afeaa0f8c7875f41f22360e6822222c0c44a1e8f699e0d922c66136113600294 not found: ID does not exist" Oct 30 12:18:05.218363 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.218344 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf6ed75b-b019-4098-83f5-85211ad2c6fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf6ed75b-b019-4098-83f5-85211ad2c6fb-tls-certs\") pod \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " Oct 30 12:18:05.218489 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.218387 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-home\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-home\") pod \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " Oct 30 12:18:05.218556 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.218546 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-dshm\") pod \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " Oct 30 12:18:05.218631 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.218618 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-model-cache\") pod \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " Oct 30 12:18:05.218686 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.218640 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6ed75b-b019-4098-83f5-85211ad2c6fb-kube-api-access-rdttq\" (UniqueName: \"kubernetes.io/projected/cf6ed75b-b019-4098-83f5-85211ad2c6fb-kube-api-access-rdttq\") pod \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " Oct 30 12:18:05.218686 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.218669 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-tmp-dir\") pod \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\" (UID: \"cf6ed75b-b019-4098-83f5-85211ad2c6fb\") " Oct 30 12:18:05.218791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.218733 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-home" pod "cf6ed75b-b019-4098-83f5-85211ad2c6fb" (UID: "cf6ed75b-b019-4098-83f5-85211ad2c6fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:18:05.218913 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.218890 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-model-cache" pod "cf6ed75b-b019-4098-83f5-85211ad2c6fb" (UID: "cf6ed75b-b019-4098-83f5-85211ad2c6fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:18:05.219019 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.218997 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-tmp-dir" pod "cf6ed75b-b019-4098-83f5-85211ad2c6fb" (UID: "cf6ed75b-b019-4098-83f5-85211ad2c6fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:18:05.219164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.219142 4413 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:18:05.219164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.219158 4413 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:18:05.219280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.219168 4413 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:18:05.220925 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.220909 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6ed75b-b019-4098-83f5-85211ad2c6fb-kube-api-access-rdttq" pod "cf6ed75b-b019-4098-83f5-85211ad2c6fb" (UID: "cf6ed75b-b019-4098-83f5-85211ad2c6fb"). InnerVolumeSpecName "kube-api-access-rdttq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:18:05.221132 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.221117 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf6ed75b-b019-4098-83f5-85211ad2c6fb-tls-certs" pod "cf6ed75b-b019-4098-83f5-85211ad2c6fb" (UID: "cf6ed75b-b019-4098-83f5-85211ad2c6fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:18:05.221176 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.221138 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-dshm" pod "cf6ed75b-b019-4098-83f5-85211ad2c6fb" (UID: "cf6ed75b-b019-4098-83f5-85211ad2c6fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:18:05.320076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.319986 4413 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf6ed75b-b019-4098-83f5-85211ad2c6fb-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:18:05.320076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.320015 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdttq\" (UniqueName: \"kubernetes.io/projected/cf6ed75b-b019-4098-83f5-85211ad2c6fb-kube-api-access-rdttq\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:18:05.320076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.320024 4413 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf6ed75b-b019-4098-83f5-85211ad2c6fb-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:18:05.539071 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.539032 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc"] Oct 30 12:18:05.548600 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:05.548571 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/e2e-conflict-a-2c88c229-kserve-8464d9bf5d-jxqhc"] Oct 30 12:18:05.813716 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:18:05.813679 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:18:06.659815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:06.659768 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:07.020426 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:07.020355 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf6ed75b-b019-4098-83f5-85211ad2c6fb" path="/var/lib/kubelet/pods/cf6ed75b-b019-4098-83f5-85211ad2c6fb/volumes" Oct 30 12:18:08.369343 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:08.369314 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:09.976430 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:09.976392 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:10.098262 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:10.098224 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:11.770848 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:11.770814 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:13.738847 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:13.738809 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:15.352165 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:15.352130 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:16.949514 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:16.949481 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:18.688833 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:18.688802 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:20.255741 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:20.255708 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:22.054669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.054634 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:22.443946 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.443911 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2"] Oct 30 12:18:22.444767 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.444748 4413 cpu_manager.go:414] "containerMap: removing container" podUID="cf6ed75b-b019-4098-83f5-85211ad2c6fb" containerName="main" Oct 30 12:18:22.444844 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.444771 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf6ed75b-b019-4098-83f5-85211ad2c6fb" containerName="main" Oct 30 12:18:22.444966 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.444954 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf6ed75b-b019-4098-83f5-85211ad2c6fb" containerName="main" Oct 30 12:18:22.450193 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.450168 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.465817 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.465793 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2"] Oct 30 12:18:22.475140 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.475121 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6a7148\"/\"distinct-w3-6a7148-kserve-self-signed-certs\"" Oct 30 12:18:22.503641 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.503605 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:22.550465 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.550439 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-dshm\") pod \"distinct-w3-6a7148-kserve-57b64c974c-5bch2\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.550602 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.550510 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsdtp\" (UniqueName: \"kubernetes.io/projected/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-kube-api-access-xsdtp\") pod \"distinct-w3-6a7148-kserve-57b64c974c-5bch2\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.550655 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.550632 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-home\") pod \"distinct-w3-6a7148-kserve-57b64c974c-5bch2\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.550789 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.550769 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-tmp-dir\") pod \"distinct-w3-6a7148-kserve-57b64c974c-5bch2\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.550866 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.550802 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-tls-certs\") pod \"distinct-w3-6a7148-kserve-57b64c974c-5bch2\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.550866 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.550855 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-model-cache\") pod \"distinct-w3-6a7148-kserve-57b64c974c-5bch2\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.651838 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.651802 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsdtp\" (UniqueName: \"kubernetes.io/projected/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-kube-api-access-xsdtp\") pod \"distinct-w3-6a7148-kserve-57b64c974c-5bch2\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.652050 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.651887 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-home\") pod \"distinct-w3-6a7148-kserve-57b64c974c-5bch2\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.652050 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.652015 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-tmp-dir\") pod \"distinct-w3-6a7148-kserve-57b64c974c-5bch2\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.652050 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.652043 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-tls-certs\") pod \"distinct-w3-6a7148-kserve-57b64c974c-5bch2\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.652331 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.652101 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-model-cache\") pod \"distinct-w3-6a7148-kserve-57b64c974c-5bch2\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.652331 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.652175 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-dshm\") pod \"distinct-w3-6a7148-kserve-57b64c974c-5bch2\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.652840 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.652601 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-tmp-dir\") pod \"distinct-w3-6a7148-kserve-57b64c974c-5bch2\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.652963 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.652850 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-home\") pod \"distinct-w3-6a7148-kserve-57b64c974c-5bch2\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.653211 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.653182 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-model-cache\") pod \"distinct-w3-6a7148-kserve-57b64c974c-5bch2\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.679272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.667610 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-dshm\") pod \"distinct-w3-6a7148-kserve-57b64c974c-5bch2\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.685488 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.685455 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsdtp\" (UniqueName: \"kubernetes.io/projected/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-kube-api-access-xsdtp\") pod \"distinct-w3-6a7148-kserve-57b64c974c-5bch2\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.686592 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.686568 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-tls-certs\") pod \"distinct-w3-6a7148-kserve-57b64c974c-5bch2\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:22.762953 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:22.762878 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:23.071468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:23.071436 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2"] Oct 30 12:18:23.072474 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:18:23.072441 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d24c3bd_af3d_4379_a8bf_7f40a42b6e90.slice/crio-00f7694d69e92e041bcf68d1439c5e1f137d65d4f39ba7a12e21d0e01991d56c WatchSource:0}: Error finding container 00f7694d69e92e041bcf68d1439c5e1f137d65d4f39ba7a12e21d0e01991d56c: Status 404 returned error can't find the container with id 00f7694d69e92e041bcf68d1439c5e1f137d65d4f39ba7a12e21d0e01991d56c Oct 30 12:18:23.363700 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:23.363618 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" event={"ID":"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90","Type":"ContainerStarted","Data":"ae72f5d2c4876472bc7957ca6a58c934cc3436fe4e99a8bb3d74ac326d8e0f41"} Oct 30 12:18:23.363700 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:23.363661 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" event={"ID":"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90","Type":"ContainerStarted","Data":"00f7694d69e92e041bcf68d1439c5e1f137d65d4f39ba7a12e21d0e01991d56c"} Oct 30 12:18:23.405428 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:23.405380 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podStartSLOduration=1.405366796 podStartE2EDuration="1.405366796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:23.403307726 +0000 UTC m=+1456.513150607" watchObservedRunningTime="2026-10-30 12:18:23.405366796 +0000 UTC m=+1456.515209671" Oct 30 12:18:23.763456 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:23.763419 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:23.894048 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:23.894015 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:25.643311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:25.643273 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:27.293167 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:27.293126 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:29.152275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:29.147099 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:31.796772 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:31.796719 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:32.763083 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:32.763056 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:32.775402 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:32.775379 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:33.569081 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:33.569052 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:18:33.969225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:33.969189 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:35.671906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:35.671876 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:35.820178 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:18:35.820136 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:18:37.441718 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:37.441684 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:39.164619 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.164585 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:39.559781 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.559701 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29"] Oct 30 12:18:39.563989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.563970 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.586734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.586709 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29"] Oct 30 12:18:39.644231 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.644206 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bebc04d-dded-466d-932f-0dfd4eec3183-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.644410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.644278 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.644410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.644351 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.644410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.644372 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.644410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.644389 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjrjg\" (UniqueName: \"kubernetes.io/projected/9bebc04d-dded-466d-932f-0dfd4eec3183-kube-api-access-mjrjg\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.644410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.644406 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.644580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.644500 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.644580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.644570 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.644642 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.644590 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.746465 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.746422 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bebc04d-dded-466d-932f-0dfd4eec3183-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.746675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.746489 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.746675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.746522 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.746675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.746550 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.746675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.746576 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjrjg\" (UniqueName: \"kubernetes.io/projected/9bebc04d-dded-466d-932f-0dfd4eec3183-kube-api-access-mjrjg\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.746675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.746602 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.746675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.746658 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.747012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.746740 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.747012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.746771 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.747120 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.747035 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.747225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.747204 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.747340 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.747299 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bebc04d-dded-466d-932f-0dfd4eec3183-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.747491 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.747472 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.747582 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.747561 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.750340 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.750317 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.751376 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.751356 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.774947 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.774917 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.775130 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.775106 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjrjg\" (UniqueName: \"kubernetes.io/projected/9bebc04d-dded-466d-932f-0dfd4eec3183-kube-api-access-mjrjg\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:39.880516 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:39.880489 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:40.197530 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.197506 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29"] Oct 30 12:18:40.198653 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:18:40.198628 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bebc04d_dded_466d_932f_0dfd4eec3183.slice/crio-76abd6c8693403b0c6bee1f5700eeba945098e507b12e15c62570591922aabe7 WatchSource:0}: Error finding container 76abd6c8693403b0c6bee1f5700eeba945098e507b12e15c62570591922aabe7: Status 404 returned error can't find the container with id 76abd6c8693403b0c6bee1f5700eeba945098e507b12e15c62570591922aabe7 Oct 30 12:18:40.201025 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.200995 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:18:40.201090 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.201063 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:18:40.201132 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.201092 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:18:40.602206 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.602119 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" event={"ID":"9bebc04d-dded-466d-932f-0dfd4eec3183","Type":"ContainerStarted","Data":"7d28111bdfaf3e767dbca32bb00d04889a7c460b236fac01b8a3e4be7a34b2dc"} Oct 30 12:18:40.602206 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.602182 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" event={"ID":"9bebc04d-dded-466d-932f-0dfd4eec3183","Type":"ContainerStarted","Data":"76abd6c8693403b0c6bee1f5700eeba945098e507b12e15c62570591922aabe7"} Oct 30 12:18:40.673447 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.673398 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" podStartSLOduration=1.673382948 podStartE2EDuration="1.673382948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:40.660774611 +0000 UTC m=+1473.770617491" watchObservedRunningTime="2026-10-30 12:18:40.673382948 +0000 UTC m=+1473.783225839" Oct 30 12:18:40.677876 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.677851 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m"] Oct 30 12:18:40.682114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.682093 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.726860 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.709499 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m"] Oct 30 12:18:40.733426 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.732879 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk"] Oct 30 12:18:40.740322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.740291 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.751569 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.751533 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk"] Oct 30 12:18:40.766297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.766038 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.766297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.766124 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpqrz\" (UniqueName: \"kubernetes.io/projected/1e53a452-5a36-4d78-9a14-35f6ebb715ba-kube-api-access-mpqrz\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.766297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.766158 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.766297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.766196 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.768933 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.766978 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.768933 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.767096 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.768933 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.767163 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.768933 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.767210 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.768933 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.767238 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.862324 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.862217 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:40.868672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.868646 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.868813 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.868688 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.868813 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.868713 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khgzc\" (UniqueName: \"kubernetes.io/projected/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-kube-api-access-khgzc\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.868937 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.868811 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqrz\" (UniqueName: \"kubernetes.io/projected/1e53a452-5a36-4d78-9a14-35f6ebb715ba-kube-api-access-mpqrz\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.868937 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.868855 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.868937 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.868906 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.869063 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.868961 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.869122 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.869095 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.869122 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.869100 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.869222 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.869195 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.869292 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.869239 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.869355 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.869284 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.869355 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.869332 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.869466 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.869380 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.869466 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.869450 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.869553 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.869536 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.869663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.869585 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.869663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.869608 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.869663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.869642 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.869821 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.869765 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.870311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.870135 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.870311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.870271 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.870458 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.870415 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.872228 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.872211 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.872566 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.872545 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.880930 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.880911 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:40.916483 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.916012 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpqrz\" (UniqueName: \"kubernetes.io/projected/1e53a452-5a36-4d78-9a14-35f6ebb715ba-kube-api-access-mpqrz\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.938519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.938343 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:40.970703 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.970670 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.970854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.970761 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.970854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.970796 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.970854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.970823 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khgzc\" (UniqueName: \"kubernetes.io/projected/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-kube-api-access-khgzc\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.970989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.970863 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.970989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.970884 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.970989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.970940 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.970989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.970982 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.971126 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.971030 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.971240 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.971213 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-workload-certs\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.971322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.971242 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-workload-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.971555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.971525 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-credential-socket\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.971669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.971535 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-data\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.971721 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.971703 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istiod-ca-cert\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.973374 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.973355 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-envoy\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:40.973583 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:40.973566 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-podinfo\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:41.002678 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.002648 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:41.009323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.004795 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khgzc\" (UniqueName: \"kubernetes.io/projected/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-kube-api-access-khgzc\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:41.019218 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.017135 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-token\") pod \"e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:41.073778 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.073746 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:41.347184 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.347153 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m"] Oct 30 12:18:41.347864 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:18:41.347838 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e53a452_5a36_4d78_9a14_35f6ebb715ba.slice/crio-b95f8ecf9c269311abefcd0529eacc79c756094f5f514a6e8eeaffe04ac04653 WatchSource:0}: Error finding container b95f8ecf9c269311abefcd0529eacc79c756094f5f514a6e8eeaffe04ac04653: Status 404 returned error can't find the container with id b95f8ecf9c269311abefcd0529eacc79c756094f5f514a6e8eeaffe04ac04653 Oct 30 12:18:41.350541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.350508 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:18:41.350618 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.350584 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:18:41.350618 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.350613 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:18:41.424950 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.424927 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk"] Oct 30 12:18:41.426970 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:18:41.426949 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b5b44c_e86c_4990_a68b_1dbff2e42a46.slice/crio-95b3a0658dfc10061b9214061a4712ffac3256ff944602d70c550e2b0661c6e6 WatchSource:0}: Error finding container 95b3a0658dfc10061b9214061a4712ffac3256ff944602d70c550e2b0661c6e6: Status 404 returned error can't find the container with id 95b3a0658dfc10061b9214061a4712ffac3256ff944602d70c550e2b0661c6e6 Oct 30 12:18:41.430223 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.430183 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:18:41.430323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.430287 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:18:41.430372 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.430331 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:18:41.630384 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.630350 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" event={"ID":"f7b5b44c-e86c-4990-a68b-1dbff2e42a46","Type":"ContainerStarted","Data":"abdd6770f5991b4437c69c662f965343b58521cf0740ecc18cb7a9a3072ce74a"} Oct 30 12:18:41.631496 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.631206 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" event={"ID":"f7b5b44c-e86c-4990-a68b-1dbff2e42a46","Type":"ContainerStarted","Data":"95b3a0658dfc10061b9214061a4712ffac3256ff944602d70c550e2b0661c6e6"} Oct 30 12:18:41.633492 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.633460 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" event={"ID":"1e53a452-5a36-4d78-9a14-35f6ebb715ba","Type":"ContainerStarted","Data":"880dba97d5d51da4be6f2467770faa1db4d132997610a073ca1367aacfbf77a6"} Oct 30 12:18:41.633602 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.633519 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" event={"ID":"1e53a452-5a36-4d78-9a14-35f6ebb715ba","Type":"ContainerStarted","Data":"b95f8ecf9c269311abefcd0529eacc79c756094f5f514a6e8eeaffe04ac04653"} Oct 30 12:18:41.675316 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.675148 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" podStartSLOduration=1.675135097 podStartE2EDuration="1.675135097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:41.674902926 +0000 UTC m=+1474.784745809" watchObservedRunningTime="2026-10-30 12:18:41.675135097 +0000 UTC m=+1474.784977978" Oct 30 12:18:41.726637 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.726563 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" podStartSLOduration=1.726542609 podStartE2EDuration="1.726542609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:41.721826207 +0000 UTC m=+1474.831669111" watchObservedRunningTime="2026-10-30 12:18:41.726542609 +0000 UTC m=+1474.836385492" Oct 30 12:18:41.882441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.882406 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 12:18:41.882637 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:41.882469 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" podUID="9bebc04d-dded-466d-932f-0dfd4eec3183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 12:18:42.003498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:42.003397 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:42.009285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:42.008761 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 12:18:42.009285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:42.008814 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" podUID="1e53a452-5a36-4d78-9a14-35f6ebb715ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 12:18:42.074086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:42.074058 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:42.075690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:42.075665 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 12:18:42.075824 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:42.075717 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" podUID="f7b5b44c-e86c-4990-a68b-1dbff2e42a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 12:18:42.677790 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:42.677753 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:42.880926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:42.880874 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 12:18:42.881132 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:42.880938 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" podUID="9bebc04d-dded-466d-932f-0dfd4eec3183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 12:18:43.007350 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:43.006703 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 12:18:43.007350 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:43.006768 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" podUID="1e53a452-5a36-4d78-9a14-35f6ebb715ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 12:18:43.074110 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:43.074078 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 12:18:43.074436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:43.074396 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" podUID="f7b5b44c-e86c-4990-a68b-1dbff2e42a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 12:18:43.881418 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:43.881380 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 12:18:43.881834 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:43.881433 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" podUID="9bebc04d-dded-466d-932f-0dfd4eec3183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 12:18:44.006849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:44.006812 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 12:18:44.007170 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:44.007124 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" podUID="1e53a452-5a36-4d78-9a14-35f6ebb715ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 12:18:44.075732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:44.075686 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 12:18:44.075923 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:44.075754 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" podUID="f7b5b44c-e86c-4990-a68b-1dbff2e42a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 12:18:44.568276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:44.566979 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:44.881106 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:44.881079 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 12:18:44.881368 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:44.881340 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" podUID="9bebc04d-dded-466d-932f-0dfd4eec3183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 12:18:45.003629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.003592 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 12:18:45.004241 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.004198 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" podUID="1e53a452-5a36-4d78-9a14-35f6ebb715ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 12:18:45.075812 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.075392 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 12:18:45.075812 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.075453 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" podUID="f7b5b44c-e86c-4990-a68b-1dbff2e42a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 12:18:45.535277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.534332 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l"] Oct 30 12:18:45.678277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.650470 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:45.680649 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.680615 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l"] Oct 30 12:18:45.680797 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.680777 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:45.746225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.721616 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"kube-root-ca.crt\"" Oct 30 12:18:45.746225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.722380 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"openshift-service-ca.crt\"" Oct 30 12:18:45.746225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.722451 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"default-dockercfg-2qxls\"" Oct 30 12:18:45.746225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.722543 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-512ec5\"/\"distinct-2-w4-512ec5-kserve-self-signed-certs\"" Oct 30 12:18:45.879335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.868534 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-dshm\") pod \"distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:45.879335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.868609 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-home\") pod \"distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:45.879335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.868636 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-tmp-dir\") pod \"distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:45.879335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.868774 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm25q\" (UniqueName: \"kubernetes.io/projected/cd22c238-a9e9-4f69-ba78-6d3992925593-kube-api-access-mm25q\") pod \"distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:45.879335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.868861 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd22c238-a9e9-4f69-ba78-6d3992925593-tls-certs\") pod \"distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:45.879335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.868918 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-model-cache\") pod \"distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:45.887276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.882449 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 12:18:45.887276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.882510 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" podUID="9bebc04d-dded-466d-932f-0dfd4eec3183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 12:18:45.983326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.975279 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd22c238-a9e9-4f69-ba78-6d3992925593-tls-certs\") pod \"distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:45.983326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.975351 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-model-cache\") pod \"distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:45.983326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.975522 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-dshm\") pod \"distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:45.983326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.975578 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-home\") pod \"distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:45.983326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.975601 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-tmp-dir\") pod \"distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:45.983326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.975725 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm25q\" (UniqueName: \"kubernetes.io/projected/cd22c238-a9e9-4f69-ba78-6d3992925593-kube-api-access-mm25q\") pod \"distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:45.983326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.983200 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-model-cache\") pod \"distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:45.983863 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.983471 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-home\") pod \"distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:45.983863 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.983694 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-tmp-dir\") pod \"distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:45.985626 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.985599 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-dshm\") pod \"distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:45.986684 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:45.986665 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd22c238-a9e9-4f69-ba78-6d3992925593-tls-certs\") pod \"distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:46.014652 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:46.014618 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 12:18:46.023351 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:46.023319 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" podUID="1e53a452-5a36-4d78-9a14-35f6ebb715ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 12:18:46.029028 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:46.028998 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm25q\" (UniqueName: \"kubernetes.io/projected/cd22c238-a9e9-4f69-ba78-6d3992925593-kube-api-access-mm25q\") pod \"distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:46.079521 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:46.079483 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 12:18:46.079740 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:46.079562 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" podUID="f7b5b44c-e86c-4990-a68b-1dbff2e42a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 12:18:46.143241 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:46.143157 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:46.550505 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:46.550425 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l"] Oct 30 12:18:46.551410 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:18:46.551370 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd22c238_a9e9_4f69_ba78_6d3992925593.slice/crio-1e5c8567591788f7735b992d80adc03bb05b33e2bd869295a42176cfe23160d3 WatchSource:0}: Error finding container 1e5c8567591788f7735b992d80adc03bb05b33e2bd869295a42176cfe23160d3: Status 404 returned error can't find the container with id 1e5c8567591788f7735b992d80adc03bb05b33e2bd869295a42176cfe23160d3 Oct 30 12:18:46.715274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:46.714832 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:46.830331 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:46.819801 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" event={"ID":"cd22c238-a9e9-4f69-ba78-6d3992925593","Type":"ContainerStarted","Data":"5ab14d6cb78aec997a9e0598a29a67541e76cec490d2fd7a6292a38b4a956c4b"} Oct 30 12:18:46.830331 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:46.819877 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" event={"ID":"cd22c238-a9e9-4f69-ba78-6d3992925593","Type":"ContainerStarted","Data":"1e5c8567591788f7735b992d80adc03bb05b33e2bd869295a42176cfe23160d3"} Oct 30 12:18:46.881014 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:46.880553 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podStartSLOduration=1.880531773 podStartE2EDuration="1.880531773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:46.879624962 +0000 UTC m=+1479.989467843" watchObservedRunningTime="2026-10-30 12:18:46.880531773 +0000 UTC m=+1479.990374657" Oct 30 12:18:46.881223 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:46.881200 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 12:18:46.881313 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:46.881285 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" podUID="9bebc04d-dded-466d-932f-0dfd4eec3183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 12:18:47.032277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:47.032020 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 12:18:47.032277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:47.032078 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" podUID="1e53a452-5a36-4d78-9a14-35f6ebb715ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 12:18:47.075283 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:47.074429 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 12:18:47.075283 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:47.074486 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" podUID="f7b5b44c-e86c-4990-a68b-1dbff2e42a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 12:18:47.145771 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:47.145747 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:48.003740 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:48.003704 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 12:18:48.004072 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:48.004032 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" podUID="1e53a452-5a36-4d78-9a14-35f6ebb715ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 12:18:48.074811 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:48.074772 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 12:18:48.075401 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:48.075371 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" podUID="f7b5b44c-e86c-4990-a68b-1dbff2e42a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 12:18:48.513850 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:48.513819 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:48.515579 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:48.515558 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:48.517845 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:48.517824 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:18:48.873782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:48.873750 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:49.037861 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:49.037637 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 12:18:49.038385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:49.038353 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" podUID="1e53a452-5a36-4d78-9a14-35f6ebb715ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 12:18:49.076685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:49.076572 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 12:18:49.076685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:49.076630 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" podUID="f7b5b44c-e86c-4990-a68b-1dbff2e42a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 12:18:50.176842 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:50.176796 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:50.177754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:50.177733 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:50.178947 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:50.178922 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:18:50.268715 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:50.268687 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:50.269488 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:50.269464 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:50.270440 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:50.270424 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:18:51.193137 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:51.193101 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:53.274482 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:53.274442 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:55.216692 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:55.216659 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:56.144434 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:56.144403 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:56.157467 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:56.157437 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:56.482865 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:56.482787 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:57.029880 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:57.029850 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:18:57.066519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:57.066486 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:57.780326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:57.780283 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:18:58.873322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:18:58.873286 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:00.632356 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:00.632316 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:01.005566 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:01.005488 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:01.136847 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:01.136813 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:01.270398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:01.270307 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:02.255275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:02.251904 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:02.379190 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:02.379155 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:02.495846 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:02.495813 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:03.569420 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:03.569363 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:03.619499 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:03.619447 4413 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" podUID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded" Oct 30 12:19:03.762276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:03.759415 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:03.762276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:03.761454 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:03.791273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:03.791216 4413 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:04.044016 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.043974 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" podUID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:04.045064 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.044303 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:04.068274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.067657 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:04.108427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.097947 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm"] Oct 30 12:19:04.155835 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.155802 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.209299 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.207446 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:04.283275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.271597 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.283275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.271706 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd9hv\" (UniqueName: \"kubernetes.io/projected/e0ce1cc0-165b-433d-88bc-719f2f95c40d-kube-api-access-zd9hv\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.283275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.271764 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.283275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.271831 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.283275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.271902 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.283275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.271938 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.283275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.272003 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.283275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.272043 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.283275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.272083 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.344332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.341338 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm"] Oct 30 12:19:04.401131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.401086 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.407471 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.407433 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.407787 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.407761 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd9hv\" (UniqueName: \"kubernetes.io/projected/e0ce1cc0-165b-433d-88bc-719f2f95c40d-kube-api-access-zd9hv\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.407963 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.407946 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.408132 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.408116 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.408331 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.408314 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.408480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.408463 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.408637 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.408622 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.408767 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.408752 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.409689 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.409636 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.410441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.410418 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.410813 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.410795 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.411169 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.411148 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.421281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.421026 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.555278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.533482 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:19:04.555278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.533548 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" podUID="9bebc04d-dded-466d-932f-0dfd4eec3183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:04.555278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.535808 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.556461 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.556394 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.599105 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.569829 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.599105 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.575446 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:04.599105 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.576322 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:19:04.718126 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.718044 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd9hv\" (UniqueName: \"kubernetes.io/projected/e0ce1cc0-165b-433d-88bc-719f2f95c40d-kube-api-access-zd9hv\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.734285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.731924 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:04.777286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.770364 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:04.777286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.770608 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 12:19:04.777286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.771674 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:19:04.777286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.771736 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:19:04.981281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.979243 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:19:04.981281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.979328 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" podUID="3a8cf876-a088-4dd2-9890-27772782da71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:04.982425 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:04.982399 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:19:05.140272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:05.127558 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" podUID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 12:19:05.447326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:05.357687 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 12:19:05.669815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:05.510743 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:05.669815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:05.576350 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:05.669815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:05.658720 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:19:05.669815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:05.667292 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:19:05.669815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:05.667369 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:19:05.669815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:05.662410 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:19:05.669815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:05.668405 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:19:05.669815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:05.668458 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:19:05.719514 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:05.719430 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm"] Oct 30 12:19:05.721692 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:05.721668 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:19:05.722509 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:05.722492 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:19:05.812654 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:05.811473 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:19:05.959815 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:19:05.959088 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:19:06.561969 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.561932 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" event={"ID":"e0ce1cc0-165b-433d-88bc-719f2f95c40d","Type":"ContainerStarted","Data":"e4cf1415150d4499116a7efaa2b874d4166ab4787e892ced60926ffa4bb7211e"} Oct 30 12:19:06.562287 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.562243 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" event={"ID":"e0ce1cc0-165b-433d-88bc-719f2f95c40d","Type":"ContainerStarted","Data":"3a89f7d2f62854685e6ad482049382b7d6fbf46f9298d60c1e9d77078832de4e"} Oct 30 12:19:06.593889 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.593853 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt"] Oct 30 12:19:06.614344 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.614309 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.615056 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.615026 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt"] Oct 30 12:19:06.631528 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.626641 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" podStartSLOduration=2.626626069 podStartE2EDuration="2.626626069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:06.622352542 +0000 UTC m=+1499.732195424" watchObservedRunningTime="2026-10-30 12:19:06.626626069 +0000 UTC m=+1499.736468951" Oct 30 12:19:06.716848 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.716814 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.717402 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.717372 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.717702 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.717678 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61e3e11-5a0b-4732-be67-e54717bc7909-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.718070 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.718044 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.718170 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.718095 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.718170 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.718132 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh992\" (UniqueName: \"kubernetes.io/projected/b61e3e11-5a0b-4732-be67-e54717bc7909-kube-api-access-rh992\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.718323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.718186 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.718819 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.718793 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.719019 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.718997 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.735272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.735211 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:06.737342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.736538 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 12:19:06.737342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.736644 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" podUID="e0ce1cc0-165b-433d-88bc-719f2f95c40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 12:19:06.822079 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.821504 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.822079 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.821728 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.822079 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.821761 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.822079 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.821989 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.824212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.822566 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61e3e11-5a0b-4732-be67-e54717bc7909-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.824212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.822631 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.824212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.822656 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.824212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.822678 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh992\" (UniqueName: \"kubernetes.io/projected/b61e3e11-5a0b-4732-be67-e54717bc7909-kube-api-access-rh992\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.824212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.822705 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.824212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.823379 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.824212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.823483 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.824212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.823111 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.824212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.823561 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.824212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.823786 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61e3e11-5a0b-4732-be67-e54717bc7909-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.825974 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.825944 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.826594 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.826572 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.846581 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.846555 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.846744 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.846614 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh992\" (UniqueName: \"kubernetes.io/projected/b61e3e11-5a0b-4732-be67-e54717bc7909-kube-api-access-rh992\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:06.935021 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:06.934971 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:07.320531 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:07.320301 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt"] Oct 30 12:19:07.324412 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:19:07.324382 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61e3e11_5a0b_4732_be67_e54717bc7909.slice/crio-f9063dae8e5e7f575ba7e0aad158733013cdf1b33029e4101b524f1c26074fd8 WatchSource:0}: Error finding container f9063dae8e5e7f575ba7e0aad158733013cdf1b33029e4101b524f1c26074fd8: Status 404 returned error can't find the container with id f9063dae8e5e7f575ba7e0aad158733013cdf1b33029e4101b524f1c26074fd8 Oct 30 12:19:07.327122 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:07.327088 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:19:07.327203 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:07.327166 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:19:07.327203 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:07.327198 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:19:07.571957 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:07.571808 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" event={"ID":"b61e3e11-5a0b-4732-be67-e54717bc7909","Type":"ContainerStarted","Data":"639bf5d108664712e8dac6ec04e6da4387b00eb774eb27cf089ba016059ee5ed"} Oct 30 12:19:07.571957 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:07.571903 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" event={"ID":"b61e3e11-5a0b-4732-be67-e54717bc7909","Type":"ContainerStarted","Data":"f9063dae8e5e7f575ba7e0aad158733013cdf1b33029e4101b524f1c26074fd8"} Oct 30 12:19:07.623785 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:07.623648 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" podStartSLOduration=1.6236306040000001 podStartE2EDuration="1.623630604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:07.621596484 +0000 UTC m=+1500.731439383" watchObservedRunningTime="2026-10-30 12:19:07.623630604 +0000 UTC m=+1500.733473486" Oct 30 12:19:07.735088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:07.735058 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 12:19:07.735620 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:07.735104 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" podUID="e0ce1cc0-165b-433d-88bc-719f2f95c40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 12:19:07.781383 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:07.781078 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:07.935529 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:07.935495 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:08.740277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:08.738926 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 12:19:08.740277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:08.738986 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" podUID="e0ce1cc0-165b-433d-88bc-719f2f95c40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 12:19:08.939750 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:08.939409 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 12:19:08.939750 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:08.939476 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" podUID="b61e3e11-5a0b-4732-be67-e54717bc7909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 12:19:09.637989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:09.637925 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:09.748989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:09.742656 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 12:19:09.748989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:09.742724 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" podUID="e0ce1cc0-165b-433d-88bc-719f2f95c40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 12:19:09.940331 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:09.939986 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 12:19:09.940331 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:09.940065 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" podUID="b61e3e11-5a0b-4732-be67-e54717bc7909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 12:19:10.744275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:10.743888 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 12:19:10.744275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:10.743954 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" podUID="e0ce1cc0-165b-433d-88bc-719f2f95c40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 12:19:10.939683 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:10.939648 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 12:19:10.940167 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:10.939813 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" podUID="b61e3e11-5a0b-4732-be67-e54717bc7909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 12:19:11.640894 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:11.640680 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:11.735788 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:11.735753 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 12:19:11.736103 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:11.736061 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" podUID="e0ce1cc0-165b-433d-88bc-719f2f95c40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 12:19:11.937535 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:11.937446 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 12:19:11.937813 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:11.937783 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" podUID="b61e3e11-5a0b-4732-be67-e54717bc7909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 12:19:12.735542 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:12.735508 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 12:19:12.736336 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:12.736291 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" podUID="e0ce1cc0-165b-433d-88bc-719f2f95c40d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 12:19:12.936436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:12.936401 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 12:19:12.936646 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:12.936458 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" podUID="b61e3e11-5a0b-4732-be67-e54717bc7909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 12:19:13.622961 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:13.622898 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:13.742999 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:13.742969 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:13.743463 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:13.743359 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:13.744080 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:13.744060 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:19:13.937978 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:13.936465 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 12:19:13.937978 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:13.936567 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" podUID="b61e3e11-5a0b-4732-be67-e54717bc7909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 12:19:14.956149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:14.956119 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:14.956819 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:14.956795 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:14.958021 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:14.957999 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:19:15.536203 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:15.536174 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:17.036781 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.036745 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:17.350206 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.350100 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:17.506973 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.506941 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr"] Oct 30 12:19:17.512454 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.512432 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.534714 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.534168 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr"] Oct 30 12:19:17.542110 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.541307 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-060f41f2-openshift-default-dockercfg-n2q4d\"" Oct 30 12:19:17.563301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.563220 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:17.637749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.636932 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-token\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.637749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.637004 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-credential-socket\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.637749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.637034 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-podinfo\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.637749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.637150 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5w52\" (UniqueName: \"kubernetes.io/projected/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-kube-api-access-g5w52\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.637749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.637200 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-envoy\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.637749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.637273 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-workload-certs\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.637749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.637375 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-workload-socket\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.637749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.637399 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istiod-ca-cert\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.637749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.637431 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-data\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.750498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.745627 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5w52\" (UniqueName: \"kubernetes.io/projected/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-kube-api-access-g5w52\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.750498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.745718 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-envoy\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.750498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.745788 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-workload-certs\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.750498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.745895 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-workload-socket\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.750498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.745921 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istiod-ca-cert\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.750498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.745957 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-data\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.750498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.746054 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-token\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.750498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.746104 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-credential-socket\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.750498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.746133 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-podinfo\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.750498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.749612 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-workload-socket\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.750498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.749928 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-workload-certs\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.750498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.750181 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-data\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.751291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.750821 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istiod-ca-cert\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.751465 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.751433 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-credential-socket\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.753913 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.753820 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-podinfo\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.766757 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.764172 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:17.777045 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.777012 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5w52\" (UniqueName: \"kubernetes.io/projected/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-kube-api-access-g5w52\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.778541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.778512 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-token\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.781544 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.781515 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-envoy\") pod \"e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:17.844006 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:17.843972 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:18.179074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:18.179046 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr"] Oct 30 12:19:18.180461 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:19:18.180434 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ec3f83_7093_4bf6_a70e_f6eab36eac9e.slice/crio-348d4d4b8dc12fa327a19af2a62292b31af4835034808f15567cb96a92ab3499 WatchSource:0}: Error finding container 348d4d4b8dc12fa327a19af2a62292b31af4835034808f15567cb96a92ab3499: Status 404 returned error can't find the container with id 348d4d4b8dc12fa327a19af2a62292b31af4835034808f15567cb96a92ab3499 Oct 30 12:19:18.182604 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:18.182570 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:19:18.182690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:18.182639 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:19:18.182690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:18.182670 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:19:18.780868 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:18.780831 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" event={"ID":"11ec3f83-7093-4bf6-a70e-f6eab36eac9e","Type":"ContainerStarted","Data":"06c3e1f4b9ccd66d35f5f81a61e14ed85286638d68edbe6e576478df61e90598"} Oct 30 12:19:18.781026 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:18.780887 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" event={"ID":"11ec3f83-7093-4bf6-a70e-f6eab36eac9e","Type":"ContainerStarted","Data":"348d4d4b8dc12fa327a19af2a62292b31af4835034808f15567cb96a92ab3499"} Oct 30 12:19:18.845038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:18.845009 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:18.849619 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:18.849572 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" podStartSLOduration=1.849558617 podStartE2EDuration="1.849558617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:18.835942178 +0000 UTC m=+1511.945785059" watchObservedRunningTime="2026-10-30 12:19:18.849558617 +0000 UTC m=+1511.959401498" Oct 30 12:19:19.069038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:19.068962 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:19.711323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:19.711289 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:19.849684 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:19.849643 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:20.794883 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:20.794852 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:20.795797 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:20.795782 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:19:20.947906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:20.947866 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:21.328769 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:21.328726 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:21.446471 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:21.446443 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:23.166519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:23.166371 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:23.421401 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:23.421300 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:23.629130 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:23.629085 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:23.855366 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:23.853672 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m"] Oct 30 12:19:23.898766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:23.898736 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" Oct 30 12:19:23.963602 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:23.963564 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-060f41f2\"" Oct 30 12:19:24.052398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:24.001840 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:24.052398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:24.041263 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-060f41f2-dockercfg-qnt6l\"" Oct 30 12:19:24.146351 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:24.126686 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 12:19:24.204275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:24.203223 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwbrd\" (UniqueName: \"kubernetes.io/projected/db4d3608-4ffc-4a96-9d5d-48d8de49facf-kube-api-access-wwbrd\") pod \"payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m\" (UID: \"db4d3608-4ffc-4a96-9d5d-48d8de49facf\") " pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" Oct 30 12:19:24.204275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:24.203472 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4d3608-4ffc-4a96-9d5d-48d8de49facf-plugins-config-volume\") pod \"payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m\" (UID: \"db4d3608-4ffc-4a96-9d5d-48d8de49facf\") " pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" Oct 30 12:19:24.305276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:24.303477 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m"] Oct 30 12:19:24.309272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:24.306688 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwbrd\" (UniqueName: \"kubernetes.io/projected/db4d3608-4ffc-4a96-9d5d-48d8de49facf-kube-api-access-wwbrd\") pod \"payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m\" (UID: \"db4d3608-4ffc-4a96-9d5d-48d8de49facf\") " pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" Oct 30 12:19:24.309272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:24.306839 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4d3608-4ffc-4a96-9d5d-48d8de49facf-plugins-config-volume\") pod \"payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m\" (UID: \"db4d3608-4ffc-4a96-9d5d-48d8de49facf\") " pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" Oct 30 12:19:24.309272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:24.307896 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4d3608-4ffc-4a96-9d5d-48d8de49facf-plugins-config-volume\") pod \"payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m\" (UID: \"db4d3608-4ffc-4a96-9d5d-48d8de49facf\") " pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" Oct 30 12:19:24.402373 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:24.394050 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwbrd\" (UniqueName: \"kubernetes.io/projected/db4d3608-4ffc-4a96-9d5d-48d8de49facf-kube-api-access-wwbrd\") pod \"payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m\" (UID: \"db4d3608-4ffc-4a96-9d5d-48d8de49facf\") " pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" Oct 30 12:19:24.621609 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:24.621563 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" Oct 30 12:19:24.809355 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:24.808350 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 12:19:24.809355 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:24.808674 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:24.809355 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:24.808844 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 12:19:24.855432 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:24.855391 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" podUID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:25.024329 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:25.004753 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:25.024930 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:25.024902 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:19:25.139185 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:25.128329 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:25.632491 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:25.462449 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:25.632491 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:25.628526 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:19:25.795408 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:25.795339 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m"] Oct 30 12:19:25.831561 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:25.831497 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 12:19:25.832344 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:25.832286 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 12:19:25.833085 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:25.833057 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:25.841332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:25.841056 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:19:25.841332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:25.841145 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:19:25.841332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:25.841205 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:19:25.892629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:25.871925 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" podUID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:25.892629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:25.872538 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:19:26.047032 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:26.046989 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 12:19:26.447104 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:26.443427 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2vh57" podUID="5a000b39-dd68-485f-8832-c6a7c7b938be" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.21:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:26.718622 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:26.718495 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:26.863827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:26.735401 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:26.863827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:26.838320 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:19:26.863827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:26.838719 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 12:19:26.863827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:26.838980 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 12:19:26.863827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:26.839273 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 12:19:26.889860 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:26.889812 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" podUID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 12:19:27.038573 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:27.038444 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" event={"ID":"db4d3608-4ffc-4a96-9d5d-48d8de49facf","Type":"ContainerStarted","Data":"7a60b549831c16c544a1c90ba1bc28c59e9a29f65dd3894037e05d8bed399d21"} Oct 30 12:19:27.091318 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:27.065873 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 12:19:27.378395 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:27.378345 4413 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded" Oct 30 12:19:27.393402 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:27.393362 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:19:27.393698 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:27.393663 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" podUID="a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:27.430025 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:27.429990 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:27.430539 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:27.430508 4413 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:27.607334 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:27.596665 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:27.861080 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:27.860979 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:27.862002 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:27.861971 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:27.862411 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:27.862388 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 12:19:27.862766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:27.862745 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 12:19:27.863079 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:27.863059 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 12:19:27.893110 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:27.892337 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" podUID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:28.180278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:28.161841 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 12:19:28.355313 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:28.355270 4413 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:28.709272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:28.700625 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:19:28.729731 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:28.729672 4413 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:28.796288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:28.776549 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:19:28.821613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:28.817595 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:19:28.845332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:28.845000 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:19:28.894296 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:28.878497 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:28.894296 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:28.879653 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 12:19:28.894296 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:28.880705 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 12:19:29.123965 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:29.091358 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:19:29.123965 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:29.091420 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:19:29.123965 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:29.091464 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:19:29.269493 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:29.269448 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" event={"ID":"db4d3608-4ffc-4a96-9d5d-48d8de49facf","Type":"ContainerStarted","Data":"32d28dafc690c0f100a7e8c2b85dba4bb0a0fb3e735d942413e6d17203e10e00"} Oct 30 12:19:29.270385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:29.269674 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" Oct 30 12:19:29.311930 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:29.311882 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" podStartSLOduration=6.311868535 podStartE2EDuration="6.311868535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:29.309626075 +0000 UTC m=+1522.419468953" watchObservedRunningTime="2026-10-30 12:19:29.311868535 +0000 UTC m=+1522.421711415" Oct 30 12:19:30.214894 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:30.214860 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:31.982721 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:31.982686 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:33.789345 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:33.789315 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:35.566590 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:35.566553 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:35.682598 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:35.682567 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:35.815067 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:19:35.815035 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:19:35.939555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:35.939515 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:36.390959 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:36.390916 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:36.502975 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:36.502944 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:37.380888 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:37.380852 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:39.153627 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:39.153593 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:40.278091 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:40.278064 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" Oct 30 12:19:40.859449 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:40.859417 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:41.738694 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:41.738633 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:42.294926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.294888 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp"] Oct 30 12:19:42.299608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.299587 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.316950 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.316927 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5649cd-openshift-default-dockercfg-h5tv6\"" Oct 30 12:19:42.333031 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.332998 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp"] Oct 30 12:19:42.378145 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.378113 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-workload-socket\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.378351 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.378178 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-workload-certs\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.378351 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.378334 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-data\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.378483 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.378366 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-token\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.378545 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.378479 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-envoy\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.378602 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.378541 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6pw4\" (UniqueName: \"kubernetes.io/projected/a3db80a9-3f95-4c9e-a252-78eca98bac38-kube-api-access-q6pw4\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.378662 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.378598 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-podinfo\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.378715 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.378660 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-credential-socket\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.378775 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.378723 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3db80a9-3f95-4c9e-a252-78eca98bac38-istiod-ca-cert\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.441793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.441765 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:42.480377 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.480356 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-envoy\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.480519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.480417 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6pw4\" (UniqueName: \"kubernetes.io/projected/a3db80a9-3f95-4c9e-a252-78eca98bac38-kube-api-access-q6pw4\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.480519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.480446 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-podinfo\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.480519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.480495 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-credential-socket\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.480687 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.480566 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3db80a9-3f95-4c9e-a252-78eca98bac38-istiod-ca-cert\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.480687 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.480635 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-workload-socket\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.480687 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.480680 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-workload-certs\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.480887 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.480776 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-data\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.480887 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.480813 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-token\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.481015 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.480936 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-credential-socket\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.481015 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.480971 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-workload-socket\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.481128 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.481063 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-workload-certs\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.481384 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.481201 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-data\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.481447 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.481396 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3db80a9-3f95-4c9e-a252-78eca98bac38-istiod-ca-cert\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.482966 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.482946 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-envoy\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.483104 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.483088 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-podinfo\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.522275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.519945 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6pw4\" (UniqueName: \"kubernetes.io/projected/a3db80a9-3f95-4c9e-a252-78eca98bac38-kube-api-access-q6pw4\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.522275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.521444 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-token\") pod \"e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.614638 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.614565 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:42.847035 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.846988 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:42.938996 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:19:42.938960 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3db80a9_3f95_4c9e_a252_78eca98bac38.slice/crio-088a6415c7967369d27a3c89721e6a4230d1d4ca17ead9e7454932a2830530b4 WatchSource:0}: Error finding container 088a6415c7967369d27a3c89721e6a4230d1d4ca17ead9e7454932a2830530b4: Status 404 returned error can't find the container with id 088a6415c7967369d27a3c89721e6a4230d1d4ca17ead9e7454932a2830530b4 Oct 30 12:19:42.940108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.940083 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp"] Oct 30 12:19:42.942344 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.942303 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:19:42.942468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.942438 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:19:42.942543 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:42.942522 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:19:43.010677 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:43.010652 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:43.369326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:43.369213 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" event={"ID":"a3db80a9-3f95-4c9e-a252-78eca98bac38","Type":"ContainerStarted","Data":"85c68c44a16bdce4eb58df2e99a37811d361bb169154881bf670786023027664"} Oct 30 12:19:43.369326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:43.369286 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" event={"ID":"a3db80a9-3f95-4c9e-a252-78eca98bac38","Type":"ContainerStarted","Data":"088a6415c7967369d27a3c89721e6a4230d1d4ca17ead9e7454932a2830530b4"} Oct 30 12:19:43.414555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:43.414506 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" podStartSLOduration=1.414492649 podStartE2EDuration="1.414492649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:43.412322008 +0000 UTC m=+1536.522164900" watchObservedRunningTime="2026-10-30 12:19:43.414492649 +0000 UTC m=+1536.524335530" Oct 30 12:19:43.614794 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:43.614760 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:44.620388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:44.620360 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:44.829739 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:44.829703 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:45.381383 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:45.381351 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:45.382436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:45.382413 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:19:45.832777 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:45.832695 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:45.951265 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:45.951221 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:47.744347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:47.744308 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf"] Oct 30 12:19:47.790278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:47.750795 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:47.790278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:47.758989 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf"] Oct 30 12:19:47.790278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:47.781516 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:47.998334 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:47.974023 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" podUID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 12:19:48.004293 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:47.999314 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 12:19:48.055907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.005422 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.055907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.005475 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.055907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.005516 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.055907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.005588 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.055907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.005639 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f079200-ed6a-4ed5-b378-9f12a141e464-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.055907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.005682 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cprf\" (UniqueName: \"kubernetes.io/projected/3f079200-ed6a-4ed5-b378-9f12a141e464-kube-api-access-2cprf\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.055907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.005766 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.055907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.005912 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.055907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.005996 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.055907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.006282 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 12:19:48.055907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.028549 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:48.055907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.029827 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:48.158864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.158827 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.159161 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.159145 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.159301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.159287 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f079200-ed6a-4ed5-b378-9f12a141e464-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.159410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.159394 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cprf\" (UniqueName: \"kubernetes.io/projected/3f079200-ed6a-4ed5-b378-9f12a141e464-kube-api-access-2cprf\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.159544 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.159532 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.159730 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.159718 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.159867 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.159854 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.159995 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.159983 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.160079 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.160069 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.172291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.160815 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f079200-ed6a-4ed5-b378-9f12a141e464-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.198819 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.198789 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.199485 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.199462 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.200199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.159868 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 12:19:48.200855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.200837 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.203354 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.201268 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.259358 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.257514 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:48.308798 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.308673 4413 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:48.334286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.329853 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.347670 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.347593 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.348167 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.348110 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.435586 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.435513 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cprf\" (UniqueName: \"kubernetes.io/projected/3f079200-ed6a-4ed5-b378-9f12a141e464-kube-api-access-2cprf\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.601521 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.601429 4413 patch_prober.go:29] interesting pod/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:19:48.601793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.601763 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" podUID="7995a1f6-7cff-4b6a-b612-78b31bad73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:48.610789 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.610524 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:48.729686 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.729629 4413 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:48.787164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.783489 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:48.787164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:48.784404 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:19:49.043827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:49.039795 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" podUID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 12:19:49.043827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:49.040324 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 12:19:49.043827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:49.040453 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:19:49.043827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:49.040510 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:19:49.043827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:49.041404 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:49.043827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:49.041673 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 12:19:49.043827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:49.042002 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:19:49.043827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:49.042054 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:19:49.043827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:49.042201 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 12:19:49.043827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:49.042489 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:19:49.220293 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:49.207132 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:49.625568 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:49.625528 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:49.795210 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:49.783872 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:49.837306 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:49.835806 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:49.935327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:49.923788 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:19:49.935327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:49.923872 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:19:49.935327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:49.923897 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984"] Oct 30 12:19:50.018277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.001732 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984"] Oct 30 12:19:50.057275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.023816 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:50.121274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.111168 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" podUID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:50.121274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.113406 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 12:19:50.121274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.115112 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:50.121274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.115649 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 12:19:50.182274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.161405 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:50.182274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.161477 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:50.182274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.161538 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:50.182274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.161602 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b9b231-747e-45be-8024-7f1b05facd48-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:50.182274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.161691 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b9b231-747e-45be-8024-7f1b05facd48-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:50.182274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.161789 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4pgb\" (UniqueName: \"kubernetes.io/projected/f9b9b231-747e-45be-8024-7f1b05facd48-kube-api-access-p4pgb\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:50.182274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.161852 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:50.182274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.161880 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:50.182274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.161905 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b9b231-747e-45be-8024-7f1b05facd48-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:50.521870 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.521579 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:50.937761 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.877891 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b9b231-747e-45be-8024-7f1b05facd48-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:50.954277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.950623 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4pgb\" (UniqueName: \"kubernetes.io/projected/f9b9b231-747e-45be-8024-7f1b05facd48-kube-api-access-p4pgb\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:50.996670 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.996633 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:50.996938 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.996914 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:51.015848 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.015814 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b9b231-747e-45be-8024-7f1b05facd48-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:51.043715 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.043683 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:51.044379 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.044354 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:51.045421 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.045399 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:51.046037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.046013 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b9b231-747e-45be-8024-7f1b05facd48-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:51.072367 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.999272 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:51.072997 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:50.998908 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:51.118990 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.118957 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b9b231-747e-45be-8024-7f1b05facd48-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:51.121286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.121235 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:51.121782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.121758 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" podUID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 12:19:51.123189 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.123165 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:51.132009 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.131463 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:51.133133 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.132966 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:51.161312 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.143941 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:51.202317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.155074 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b9b231-747e-45be-8024-7f1b05facd48-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:51.202317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.156513 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:19:51.202317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.190192 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:19:51.202317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.190273 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:19:51.202317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.164358 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:51.202317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.164383 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 12:19:51.202317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.201501 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:51.345693 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.341628 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4pgb\" (UniqueName: \"kubernetes.io/projected/f9b9b231-747e-45be-8024-7f1b05facd48-kube-api-access-p4pgb\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:51.418282 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.400854 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b9b231-747e-45be-8024-7f1b05facd48-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:51.546450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.544648 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:51.619150 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.619120 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf"] Oct 30 12:19:51.686278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.684743 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:51.694284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:51.691472 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" event={"ID":"3f079200-ed6a-4ed5-b378-9f12a141e464","Type":"ContainerStarted","Data":"80874ea33f7489c714f825db9868275aed1179de04b1c2b15b43f21d4014d12a"} Oct 30 12:19:52.005678 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:52.004990 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:52.124486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:52.123683 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" podUID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:52.133641 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:52.133311 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:52.134581 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:52.134295 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:52.162424 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:52.162324 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 12:19:52.203390 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:52.203345 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:52.271776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:52.203644 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:52.272007 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:52.271988 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:19:52.273693 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:52.273676 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 12:19:52.273868 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:52.273853 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:19:52.563329 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:52.544893 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:52.884002 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:52.883909 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984"] Oct 30 12:19:53.064047 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:52.932997 4413 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:53.179961 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:53.176428 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:53.179961 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:53.177036 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 12:19:53.276523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:53.272566 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:19:53.276523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:53.272651 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:19:53.276523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:53.272695 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:19:53.276523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:53.275423 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:19:53.276523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:53.276023 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 12:19:53.337312 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:53.336010 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" event={"ID":"3f079200-ed6a-4ed5-b378-9f12a141e464","Type":"ContainerStarted","Data":"930ce6b1eead00224d7604c304066552d5df3ac5e7a0c5f20624d96aabfb3340"} Oct 30 12:19:53.337312 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:53.336107 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 12:19:53.340143 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:53.340122 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:19:53.353299 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:53.345511 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:19:53.371295 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:53.368944 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" podStartSLOduration=6.368923576 podStartE2EDuration="6.368923576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:53.35589685 +0000 UTC m=+1546.465739732" watchObservedRunningTime="2026-10-30 12:19:53.368923576 +0000 UTC m=+1546.478766461" Oct 30 12:19:53.384886 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:53.384834 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 12:19:53.481622 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:53.481523 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 12:19:53.623289 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:53.621062 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:19:53.631458 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:53.630464 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 12:19:53.631458 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:53.630527 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" podUID="3f079200-ed6a-4ed5-b378-9f12a141e464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 12:19:54.255878 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:54.255834 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:54.265427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:54.265068 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" event={"ID":"f9b9b231-747e-45be-8024-7f1b05facd48","Type":"ContainerStarted","Data":"3a20a90e50605927a6d28c38dbe5075b465e29d260b2742e3f0fd739e00ea3ff"} Oct 30 12:19:54.265427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:54.265131 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" event={"ID":"f9b9b231-747e-45be-8024-7f1b05facd48","Type":"ContainerStarted","Data":"9e46d20dce12a20345352bcf6f0a13f85e8b404a2e25fbeee85c83203254ce63"} Oct 30 12:19:54.318280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:54.316308 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" podStartSLOduration=5.31629025 podStartE2EDuration="5.31629025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:54.315889783 +0000 UTC m=+1547.425732713" watchObservedRunningTime="2026-10-30 12:19:54.31629025 +0000 UTC m=+1547.426133156" Oct 30 12:19:54.611719 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:54.611634 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 12:19:54.611962 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:54.611929 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" podUID="3f079200-ed6a-4ed5-b378-9f12a141e464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 12:19:54.686191 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:54.686155 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:19:54.687662 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:54.687640 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 12:19:54.687754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:54.687693 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" podUID="f9b9b231-747e-45be-8024-7f1b05facd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 12:19:55.612664 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:55.612625 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 12:19:55.613175 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:55.612688 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" podUID="3f079200-ed6a-4ed5-b378-9f12a141e464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 12:19:55.686459 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:55.686429 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 12:19:55.686686 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:55.686658 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" podUID="f9b9b231-747e-45be-8024-7f1b05facd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 12:19:56.065334 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:56.065299 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:56.612276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:56.611751 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 12:19:56.612276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:56.611833 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" podUID="3f079200-ed6a-4ed5-b378-9f12a141e464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 12:19:56.644122 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:56.644086 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:56.686559 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:56.686531 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 12:19:56.686704 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:56.686581 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" podUID="f9b9b231-747e-45be-8024-7f1b05facd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 12:19:56.874503 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:56.874474 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:57.619715 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:57.619680 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 12:19:57.619983 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:57.619955 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" podUID="3f079200-ed6a-4ed5-b378-9f12a141e464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 12:19:57.686114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:57.686082 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 12:19:57.686574 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:57.686549 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" podUID="f9b9b231-747e-45be-8024-7f1b05facd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 12:19:57.818893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:57.818815 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:58.613086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:58.612633 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 12:19:58.613086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:58.612698 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" podUID="3f079200-ed6a-4ed5-b378-9f12a141e464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 12:19:58.686280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:58.686222 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 12:19:58.686987 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:58.686691 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" podUID="f9b9b231-747e-45be-8024-7f1b05facd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 12:19:59.479520 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:59.479480 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:19:59.628710 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:59.628656 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 12:19:59.628900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:59.628727 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" podUID="3f079200-ed6a-4ed5-b378-9f12a141e464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 12:19:59.686335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:59.686302 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 12:19:59.686709 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:19:59.686367 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" podUID="f9b9b231-747e-45be-8024-7f1b05facd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 12:20:00.616688 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:00.616652 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:20:00.617114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:00.617097 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:20:00.618141 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:00.618120 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:20:00.686096 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:00.686050 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 12:20:00.686290 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:00.686127 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" podUID="f9b9b231-747e-45be-8024-7f1b05facd48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 12:20:01.111120 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:01.111009 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:01.227291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:01.227101 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:01.338883 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:01.338853 4413 generic.go:356] "Generic (PLEG): container finished" podID="11ec3f83-7093-4bf6-a70e-f6eab36eac9e" containerID="06c3e1f4b9ccd66d35f5f81a61e14ed85286638d68edbe6e576478df61e90598" exitCode=0 Oct 30 12:20:01.339094 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:01.338929 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" event={"ID":"11ec3f83-7093-4bf6-a70e-f6eab36eac9e","Type":"ContainerDied","Data":"06c3e1f4b9ccd66d35f5f81a61e14ed85286638d68edbe6e576478df61e90598"} Oct 30 12:20:01.339503 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:01.339485 4413 scope.go:122] "RemoveContainer" containerID="06c3e1f4b9ccd66d35f5f81a61e14ed85286638d68edbe6e576478df61e90598" Oct 30 12:20:01.340878 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:01.340851 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:20:01.340937 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:01.340919 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:20:01.340977 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:01.340955 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:20:01.711271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:01.711096 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:20:01.712486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:01.711779 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:20:01.713077 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:01.713054 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 12:20:02.366343 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:02.365487 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" event={"ID":"11ec3f83-7093-4bf6-a70e-f6eab36eac9e","Type":"ContainerStarted","Data":"29dda7754af5fae597fc14f4c2ed1c939c465fd0e5b210968e86401af155684c"} Oct 30 12:20:02.845161 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:02.845134 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:20:02.846055 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:02.846029 4413 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 12:20:02.846185 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:02.846086 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" podUID="11ec3f83-7093-4bf6-a70e-f6eab36eac9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 12:20:03.225945 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:03.225909 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:03.844605 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:03.844572 4413 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 12:20:03.845013 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:03.844641 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" podUID="11ec3f83-7093-4bf6-a70e-f6eab36eac9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 12:20:04.844883 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:04.844844 4413 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 12:20:04.845321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:04.844918 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" podUID="11ec3f83-7093-4bf6-a70e-f6eab36eac9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 12:20:05.021736 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:05.021694 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:05.815635 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:05.815584 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:20:05.840745 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:20:05.840708 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:20:05.845333 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:05.845295 4413 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 12:20:05.845731 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:05.845364 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" podUID="11ec3f83-7093-4bf6-a70e-f6eab36eac9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 12:20:06.733420 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:06.733366 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:06.845240 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:06.845196 4413 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 12:20:06.845486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:06.845303 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" podUID="11ec3f83-7093-4bf6-a70e-f6eab36eac9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 12:20:07.242515 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:07.242483 4413 scope.go:122] "RemoveContainer" containerID="9ccf40639de88ad19f51f641d2b3337e73d508bf00e005388b073ee92e024a25" Oct 30 12:20:07.952274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:07.947983 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:20:07.972277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:07.968589 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:20:08.735921 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:08.735881 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:10.559020 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:10.558964 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:10.902272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:10.902217 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5"] Oct 30 12:20:10.907147 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:10.907110 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:10.925760 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:10.925722 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5"] Oct 30 12:20:10.933570 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:10.933536 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-060f41f2\"/\"default-dockercfg-jgjpl\"" Oct 30 12:20:10.933786 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:10.933611 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-060f41f2\"/\"kube-root-ca.crt\"" Oct 30 12:20:10.933786 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:10.933534 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-060f41f2\"/\"e2e-lifecycle-model-060f41f2-kserve-self-signed-certs\"" Oct 30 12:20:10.933786 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:10.933655 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-060f41f2\"/\"openshift-service-ca.crt\"" Oct 30 12:20:10.968407 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:10.968369 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:10.991088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:10.991054 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-dshm\") pod \"e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:10.991323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:10.991108 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-home\") pod \"e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:10.991323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:10.991139 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-tmp-dir\") pod \"e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:10.991323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:10.991232 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-tls-certs\") pod \"e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:10.991496 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:10.991343 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk4fj\" (UniqueName: \"kubernetes.io/projected/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-kube-api-access-lk4fj\") pod \"e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:10.991496 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:10.991388 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-model-cache\") pod \"e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:11.092827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:11.092787 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-home\") pod \"e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:11.093014 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:11.092843 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-tmp-dir\") pod \"e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:11.093014 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:11.092862 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-tls-certs\") pod \"e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:11.093014 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:11.092927 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk4fj\" (UniqueName: \"kubernetes.io/projected/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-kube-api-access-lk4fj\") pod \"e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:11.093159 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:11.093068 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-model-cache\") pod \"e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:11.093402 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:11.093366 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-tmp-dir\") pod \"e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:11.093402 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:11.093366 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-home\") pod \"e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:11.093548 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:11.093452 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-dshm\") pod \"e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:11.093548 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:11.093455 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-model-cache\") pod \"e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:11.095632 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:11.095610 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-dshm\") pod \"e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:11.095716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:11.095652 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-tls-certs\") pod \"e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:11.116438 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:11.116404 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk4fj\" (UniqueName: \"kubernetes.io/projected/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-kube-api-access-lk4fj\") pod \"e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:11.229430 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:11.229347 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:11.548922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:11.548894 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5"] Oct 30 12:20:11.549740 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:20:11.549715 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86a3a7f8_9d16_47d3_bb8d_15f4b4d4b183.slice/crio-6a1e3e4a7f00e2d7595edeafd5f9f54a98de4d5b35350e2aad87c0568a914fd2 WatchSource:0}: Error finding container 6a1e3e4a7f00e2d7595edeafd5f9f54a98de4d5b35350e2aad87c0568a914fd2: Status 404 returned error can't find the container with id 6a1e3e4a7f00e2d7595edeafd5f9f54a98de4d5b35350e2aad87c0568a914fd2 Oct 30 12:20:12.371137 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:12.371101 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:12.446636 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:12.446600 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" event={"ID":"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183","Type":"ContainerStarted","Data":"36083e1551e2caafbe1076dfe8566495a6514df884d67614ef57274702001629"} Oct 30 12:20:12.446836 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:12.446681 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" event={"ID":"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183","Type":"ContainerStarted","Data":"6a1e3e4a7f00e2d7595edeafd5f9f54a98de4d5b35350e2aad87c0568a914fd2"} Oct 30 12:20:12.486519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:12.486415 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" podStartSLOduration=2.486401604 podStartE2EDuration="2.486401604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:12.486190621 +0000 UTC m=+1565.596033502" watchObservedRunningTime="2026-10-30 12:20:12.486401604 +0000 UTC m=+1565.596244484" Oct 30 12:20:13.229848 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:13.229812 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:14.213233 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:14.213186 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:15.762309 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:15.762209 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:17.440384 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:17.440350 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:19.266551 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:19.266473 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:21.001563 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:21.001531 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:21.230012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:21.229982 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:21.242384 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:21.242359 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:21.518564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:21.518537 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:20:21.573001 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:21.572965 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:21.818428 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:21.818340 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n"] Oct 30 12:20:21.823072 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:21.823046 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:21.858566 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:21.858525 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n"] Oct 30 12:20:21.975446 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:21.975411 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:21.975446 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:21.975449 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d6k7\" (UniqueName: \"kubernetes.io/projected/9f031740-a3c2-413b-9943-8d2d31f143a0-kube-api-access-5d6k7\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:21.975673 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:21.975482 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:21.975673 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:21.975590 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:21.975673 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:21.975616 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f031740-a3c2-413b-9943-8d2d31f143a0-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:21.975818 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:21.975806 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:21.975861 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:21.975850 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:21.975932 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:21.975914 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:21.975968 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:21.975942 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.077819 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.077734 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.077819 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.077807 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.078387 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.077848 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.078387 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.077876 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.078387 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.077934 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.078387 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.077957 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d6k7\" (UniqueName: \"kubernetes.io/projected/9f031740-a3c2-413b-9943-8d2d31f143a0-kube-api-access-5d6k7\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.078387 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.077997 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.078387 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.078036 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.078387 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.078074 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f031740-a3c2-413b-9943-8d2d31f143a0-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.078387 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.078299 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-workload-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.078697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.078402 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-workload-certs\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.078697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.078402 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-credential-socket\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.078697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.078599 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-data\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.079079 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.079055 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f031740-a3c2-413b-9943-8d2d31f143a0-istiod-ca-cert\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.080710 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.080690 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-envoy\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.080790 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.080722 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-podinfo\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.103137 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.103098 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-token\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.103305 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.103263 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d6k7\" (UniqueName: \"kubernetes.io/projected/9f031740-a3c2-413b-9943-8d2d31f143a0-kube-api-access-5d6k7\") pod \"e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.138225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.138195 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:22.354020 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.353938 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:22.472570 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.472539 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n"] Oct 30 12:20:22.473060 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:20:22.473029 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f031740_a3c2_413b_9943_8d2d31f143a0.slice/crio-ad6f28b212fa9cf35d208cecf26a735ab39db1ce025e0e2d905fbd82486660b2 WatchSource:0}: Error finding container ad6f28b212fa9cf35d208cecf26a735ab39db1ce025e0e2d905fbd82486660b2: Status 404 returned error can't find the container with id ad6f28b212fa9cf35d208cecf26a735ab39db1ce025e0e2d905fbd82486660b2 Oct 30 12:20:22.475846 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.475807 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:20:22.475947 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.475898 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:20:22.475989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.475940 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:20:22.520018 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.519969 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" event={"ID":"9f031740-a3c2-413b-9943-8d2d31f143a0","Type":"ContainerStarted","Data":"ad6f28b212fa9cf35d208cecf26a735ab39db1ce025e0e2d905fbd82486660b2"} Oct 30 12:20:22.868690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:22.868588 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:23.018338 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:23.016982 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:23.471274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:23.471064 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:23.551288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:23.549816 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" event={"ID":"9f031740-a3c2-413b-9943-8d2d31f143a0","Type":"ContainerStarted","Data":"4f1f002f9cd2279eb36b3c6fa3cf6fb926ea09fecd5c86a7e688a97c5d99910c"} Oct 30 12:20:23.790344 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:23.784665 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" podStartSLOduration=2.784639744 podStartE2EDuration="2.784639744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:23.643198909 +0000 UTC m=+1576.753041791" watchObservedRunningTime="2026-10-30 12:20:23.784639744 +0000 UTC m=+1576.894482626" Oct 30 12:20:24.221274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:24.220483 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:24.313277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:24.301309 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:20:24.313277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:24.301377 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" podUID="9f031740-a3c2-413b-9943-8d2d31f143a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 12:20:25.142719 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:25.142681 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:20:25.143359 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:25.143314 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" podUID="9f031740-a3c2-413b-9943-8d2d31f143a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 12:20:25.261620 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:25.261577 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:26.142326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:26.138795 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:20:26.142326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:26.138858 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" podUID="9f031740-a3c2-413b-9943-8d2d31f143a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 12:20:27.145276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:27.142772 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:20:27.145276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:27.142841 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" podUID="9f031740-a3c2-413b-9943-8d2d31f143a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 12:20:27.173436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:27.173402 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:27.286556 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:27.286522 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:27.482593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:27.482491 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:27.677463 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:27.677427 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:28.141659 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:28.141616 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:20:28.141991 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:28.141960 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" podUID="9f031740-a3c2-413b-9943-8d2d31f143a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 12:20:29.067009 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:29.066980 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:29.140286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:29.140205 4413 patch_prober.go:29] interesting pod/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 12:20:29.140472 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:29.140310 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" podUID="9f031740-a3c2-413b-9943-8d2d31f143a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 12:20:29.182746 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:29.182700 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:29.225829 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:29.225798 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll"] Oct 30 12:20:29.229971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:29.229949 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll" Oct 30 12:20:29.248265 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:29.248220 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll"] Oct 30 12:20:29.257286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:29.255152 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7566f5-dockercfg-nsvjc\"" Oct 30 12:20:29.257286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:29.255152 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7566f5\"" Oct 30 12:20:29.381273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:29.378772 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5265e74-61ce-4747-a4b0-df93e28a83cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll\" (UID: \"f5265e74-61ce-4747-a4b0-df93e28a83cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll" Oct 30 12:20:29.399339 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:29.393570 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6vlm\" (UniqueName: \"kubernetes.io/projected/f5265e74-61ce-4747-a4b0-df93e28a83cc-kube-api-access-n6vlm\") pod \"payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll\" (UID: \"f5265e74-61ce-4747-a4b0-df93e28a83cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll" Oct 30 12:20:29.526141 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:29.526106 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5265e74-61ce-4747-a4b0-df93e28a83cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll\" (UID: \"f5265e74-61ce-4747-a4b0-df93e28a83cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll" Oct 30 12:20:29.526358 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:29.526199 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6vlm\" (UniqueName: \"kubernetes.io/projected/f5265e74-61ce-4747-a4b0-df93e28a83cc-kube-api-access-n6vlm\") pod \"payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll\" (UID: \"f5265e74-61ce-4747-a4b0-df93e28a83cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll" Oct 30 12:20:29.527131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:29.527103 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5265e74-61ce-4747-a4b0-df93e28a83cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll\" (UID: \"f5265e74-61ce-4747-a4b0-df93e28a83cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll" Oct 30 12:20:29.549698 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:29.549643 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6vlm\" (UniqueName: \"kubernetes.io/projected/f5265e74-61ce-4747-a4b0-df93e28a83cc-kube-api-access-n6vlm\") pod \"payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll\" (UID: \"f5265e74-61ce-4747-a4b0-df93e28a83cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll" Oct 30 12:20:29.849189 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:29.849084 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll" Oct 30 12:20:30.163392 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:30.163360 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll"] Oct 30 12:20:30.638991 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:30.638952 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:30.639403 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:30.639375 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:30.640196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:30.640172 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:20:30.679918 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:30.679873 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll" event={"ID":"f5265e74-61ce-4747-a4b0-df93e28a83cc","Type":"ContainerStarted","Data":"f06544cae3910e4647766a45b6513e45c8ae7f59e8a52626d097203042af3c34"} Oct 30 12:20:30.680097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:30.679950 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll" event={"ID":"f5265e74-61ce-4747-a4b0-df93e28a83cc","Type":"ContainerStarted","Data":"2441878cc8c478edf2d44760efbd8372d849123b08517188508fd6d3cd8227a4"} Oct 30 12:20:30.680097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:30.679980 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll" Oct 30 12:20:30.780343 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:30.780273 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll" podStartSLOduration=1.780241406 podStartE2EDuration="1.780241406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:30.778198982 +0000 UTC m=+1583.888041863" watchObservedRunningTime="2026-10-30 12:20:30.780241406 +0000 UTC m=+1583.890084286" Oct 30 12:20:30.947979 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:30.947879 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:31.423426 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:31.423380 4413 patch_prober.go:29] interesting pod/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:20:31.423922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:31.423451 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" podUID="a3db80a9-3f95-4c9e-a252-78eca98bac38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:20:32.195451 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.195399 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:32.635459 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.635423 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9"] Oct 30 12:20:32.640463 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.640441 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.660938 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.660906 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-82299a-openshift-default-dockercfg-qtvkc\"" Oct 30 12:20:32.675513 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.675479 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9"] Oct 30 12:20:32.682636 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.682541 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:32.787814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.786854 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-token\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.787814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.787009 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-workload-socket\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.787814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.787216 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-credential-socket\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.787814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.787287 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-data\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.787814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.787369 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-podinfo\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.787814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.787433 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-workload-certs\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.787814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.787517 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istiod-ca-cert\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.787814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.787560 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s77pc\" (UniqueName: \"kubernetes.io/projected/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-kube-api-access-s77pc\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.787814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.787692 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-envoy\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.814583 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.797885 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:32.889385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.889125 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-podinfo\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.889385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.889191 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-workload-certs\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.889385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.889327 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istiod-ca-cert\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.889385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.889369 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s77pc\" (UniqueName: \"kubernetes.io/projected/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-kube-api-access-s77pc\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.889864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.889457 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-envoy\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.889864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.889519 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-token\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.889864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.889620 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-workload-socket\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.889864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.889761 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-credential-socket\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.889864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.889807 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-data\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.891828 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.890303 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-data\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.891828 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.891277 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-workload-socket\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.892027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.891939 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-credential-socket\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.893742 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.893691 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-workload-certs\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.894845 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.894231 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istiod-ca-cert\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.894845 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.894793 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-podinfo\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.901945 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.901909 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-envoy\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.933706 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.929148 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-token\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.933706 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.931742 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s77pc\" (UniqueName: \"kubernetes.io/projected/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-kube-api-access-s77pc\") pod \"e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:32.933706 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.931938 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:32.956881 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:32.956851 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:33.300667 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:33.300644 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9"] Oct 30 12:20:33.301299 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:20:33.301235 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d4ab61_5cb6_4c3f_9692_ece09ff0bd3d.slice/crio-8678aab983f6272efca3057f6b075f3aeafd14482599b179928de324ece5b7bd WatchSource:0}: Error finding container 8678aab983f6272efca3057f6b075f3aeafd14482599b179928de324ece5b7bd: Status 404 returned error can't find the container with id 8678aab983f6272efca3057f6b075f3aeafd14482599b179928de324ece5b7bd Oct 30 12:20:33.303887 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:33.303852 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:20:33.303960 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:33.303929 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:20:33.304000 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:33.303963 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:20:33.707558 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:33.707519 4413 generic.go:356] "Generic (PLEG): container finished" podID="a3db80a9-3f95-4c9e-a252-78eca98bac38" containerID="85c68c44a16bdce4eb58df2e99a37811d361bb169154881bf670786023027664" exitCode=0 Oct 30 12:20:33.708041 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:33.707587 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" event={"ID":"a3db80a9-3f95-4c9e-a252-78eca98bac38","Type":"ContainerDied","Data":"85c68c44a16bdce4eb58df2e99a37811d361bb169154881bf670786023027664"} Oct 30 12:20:33.708129 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:33.708044 4413 scope.go:122] "RemoveContainer" containerID="85c68c44a16bdce4eb58df2e99a37811d361bb169154881bf670786023027664" Oct 30 12:20:33.709508 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:33.709482 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" event={"ID":"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d","Type":"ContainerStarted","Data":"97b96593d501ed9d9131cdf7055ed4baa97c4a68c19b584b1256cb90725bd513"} Oct 30 12:20:33.709601 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:33.709534 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" event={"ID":"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d","Type":"ContainerStarted","Data":"8678aab983f6272efca3057f6b075f3aeafd14482599b179928de324ece5b7bd"} Oct 30 12:20:33.709601 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:33.709579 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:20:33.709686 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:33.709647 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:20:33.709720 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:33.709690 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:20:33.821235 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:33.821182 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" podStartSLOduration=1.821166394 podStartE2EDuration="1.821166394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:33.818374849 +0000 UTC m=+1586.928217731" watchObservedRunningTime="2026-10-30 12:20:33.821166394 +0000 UTC m=+1586.931009276" Oct 30 12:20:33.957646 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:33.957542 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:34.481585 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:34.481550 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:34.727547 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:34.727511 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" event={"ID":"a3db80a9-3f95-4c9e-a252-78eca98bac38","Type":"ContainerStarted","Data":"27a457f1228bcd4388bea456c05fd502acace256d3c33b6bfe9d545293c15cab"} Oct 30 12:20:34.963020 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:34.962990 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:35.615437 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:35.615399 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:20:35.615652 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:35.615626 4413 patch_prober.go:29] interesting pod/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 12:20:35.615731 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:35.615704 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" podUID="a3db80a9-3f95-4c9e-a252-78eca98bac38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 12:20:35.734158 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:35.734118 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:35.735204 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:35.735189 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:20:35.859792 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:20:35.859755 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:20:36.315549 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:36.314170 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:36.432140 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:36.432111 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:36.619334 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:36.616644 4413 patch_prober.go:29] interesting pod/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 12:20:36.619334 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:36.616726 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" podUID="a3db80a9-3f95-4c9e-a252-78eca98bac38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 12:20:37.615709 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:37.615675 4413 patch_prober.go:29] interesting pod/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 12:20:37.616220 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:37.615739 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" podUID="a3db80a9-3f95-4c9e-a252-78eca98bac38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 12:20:38.021466 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:38.021416 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:38.140372 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:38.140339 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:38.261181 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:38.261143 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:38.621758 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:38.621719 4413 patch_prober.go:29] interesting pod/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 12:20:38.622166 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:38.621796 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" podUID="a3db80a9-3f95-4c9e-a252-78eca98bac38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 12:20:39.615723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:39.615689 4413 patch_prober.go:29] interesting pod/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 12:20:39.615912 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:39.615759 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" podUID="a3db80a9-3f95-4c9e-a252-78eca98bac38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 12:20:40.186445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:40.186412 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:40.618984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:40.618904 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:20:40.619454 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:40.619437 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:20:40.620395 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:40.620370 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:20:41.688997 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.688971 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll" Oct 30 12:20:41.729267 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.729224 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb"] Oct 30 12:20:41.733729 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.733700 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" Oct 30 12:20:41.750981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.750952 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-7566f5-serving-cert\"" Oct 30 12:20:41.757619 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.757589 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb"] Oct 30 12:20:41.783851 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.783810 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfxjk\" (UniqueName: \"kubernetes.io/projected/50dada57-ddc4-4a37-bdd5-638e44f15a63-kube-api-access-sfxjk\") pod \"maas-api-e2e-api-b-7566f5-6d88789898-7t2nb\" (UID: \"50dada57-ddc4-4a37-bdd5-638e44f15a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" Oct 30 12:20:41.784217 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.784193 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dada57-ddc4-4a37-bdd5-638e44f15a63-metrics-certs\") pod \"maas-api-e2e-api-b-7566f5-6d88789898-7t2nb\" (UID: \"50dada57-ddc4-4a37-bdd5-638e44f15a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" Oct 30 12:20:41.784592 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.784565 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50dada57-ddc4-4a37-bdd5-638e44f15a63-maas-api-tls\") pod \"maas-api-e2e-api-b-7566f5-6d88789898-7t2nb\" (UID: \"50dada57-ddc4-4a37-bdd5-638e44f15a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" Oct 30 12:20:41.784908 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.784884 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50dada57-ddc4-4a37-bdd5-638e44f15a63-openshift-service-ca\") pod \"maas-api-e2e-api-b-7566f5-6d88789898-7t2nb\" (UID: \"50dada57-ddc4-4a37-bdd5-638e44f15a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" Oct 30 12:20:41.809404 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.809360 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:41.887093 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.887055 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfxjk\" (UniqueName: \"kubernetes.io/projected/50dada57-ddc4-4a37-bdd5-638e44f15a63-kube-api-access-sfxjk\") pod \"maas-api-e2e-api-b-7566f5-6d88789898-7t2nb\" (UID: \"50dada57-ddc4-4a37-bdd5-638e44f15a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" Oct 30 12:20:41.887269 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.887192 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dada57-ddc4-4a37-bdd5-638e44f15a63-metrics-certs\") pod \"maas-api-e2e-api-b-7566f5-6d88789898-7t2nb\" (UID: \"50dada57-ddc4-4a37-bdd5-638e44f15a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" Oct 30 12:20:41.887341 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.887286 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50dada57-ddc4-4a37-bdd5-638e44f15a63-maas-api-tls\") pod \"maas-api-e2e-api-b-7566f5-6d88789898-7t2nb\" (UID: \"50dada57-ddc4-4a37-bdd5-638e44f15a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" Oct 30 12:20:41.887456 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.887436 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50dada57-ddc4-4a37-bdd5-638e44f15a63-openshift-service-ca\") pod \"maas-api-e2e-api-b-7566f5-6d88789898-7t2nb\" (UID: \"50dada57-ddc4-4a37-bdd5-638e44f15a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" Oct 30 12:20:41.888555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.888525 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50dada57-ddc4-4a37-bdd5-638e44f15a63-openshift-service-ca\") pod \"maas-api-e2e-api-b-7566f5-6d88789898-7t2nb\" (UID: \"50dada57-ddc4-4a37-bdd5-638e44f15a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" Oct 30 12:20:41.890240 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.890194 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dada57-ddc4-4a37-bdd5-638e44f15a63-metrics-certs\") pod \"maas-api-e2e-api-b-7566f5-6d88789898-7t2nb\" (UID: \"50dada57-ddc4-4a37-bdd5-638e44f15a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" Oct 30 12:20:41.890392 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.890371 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50dada57-ddc4-4a37-bdd5-638e44f15a63-maas-api-tls\") pod \"maas-api-e2e-api-b-7566f5-6d88789898-7t2nb\" (UID: \"50dada57-ddc4-4a37-bdd5-638e44f15a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" Oct 30 12:20:41.910376 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.910355 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfxjk\" (UniqueName: \"kubernetes.io/projected/50dada57-ddc4-4a37-bdd5-638e44f15a63-kube-api-access-sfxjk\") pod \"maas-api-e2e-api-b-7566f5-6d88789898-7t2nb\" (UID: \"50dada57-ddc4-4a37-bdd5-638e44f15a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" Oct 30 12:20:41.921557 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:41.921529 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:42.050894 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:42.050816 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" Oct 30 12:20:42.365871 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:42.365842 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb"] Oct 30 12:20:42.366911 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:20:42.366881 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50dada57_ddc4_4a37_bdd5_638e44f15a63.slice/crio-52c357bc441bf59f40c2ebd3132be6bb1b3cb10444bb5f6a71c261d7b9978b3c WatchSource:0}: Error finding container 52c357bc441bf59f40c2ebd3132be6bb1b3cb10444bb5f6a71c261d7b9978b3c: Status 404 returned error can't find the container with id 52c357bc441bf59f40c2ebd3132be6bb1b3cb10444bb5f6a71c261d7b9978b3c Oct 30 12:20:42.368427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:42.368404 4413 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:20:42.795616 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:42.795591 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" event={"ID":"50dada57-ddc4-4a37-bdd5-638e44f15a63","Type":"ContainerStarted","Data":"52c357bc441bf59f40c2ebd3132be6bb1b3cb10444bb5f6a71c261d7b9978b3c"} Oct 30 12:20:43.741381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:43.741351 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:43.812379 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:43.812333 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" event={"ID":"50dada57-ddc4-4a37-bdd5-638e44f15a63","Type":"ContainerStarted","Data":"1c53e35dda16cea704270a009ae7fcfb50c40c5a6300f3143e666232146dcaea"} Oct 30 12:20:43.812871 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:43.812467 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" Oct 30 12:20:43.856847 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:43.856794 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" podStartSLOduration=2.489668686 podStartE2EDuration="2.856779138s" totalImagesPullingTime="367.110452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:43.854820965 +0000 UTC m=+1596.964663845" watchObservedRunningTime="2026-10-30 12:20:43.856779138 +0000 UTC m=+1596.966622071" Oct 30 12:20:45.366035 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:45.365991 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:47.262774 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:47.262741 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:47.918462 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:47.918428 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx"] Oct 30 12:20:47.923648 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:47.923623 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:47.962697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:47.962661 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx"] Oct 30 12:20:48.065944 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.065905 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d871b22-0655-4cc4-9afb-686f9429693f-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.066151 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.065979 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.066151 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.066091 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.066273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.066169 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.066273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.066199 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d871b22-0655-4cc4-9afb-686f9429693f-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.066358 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.066288 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d871b22-0655-4cc4-9afb-686f9429693f-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.066358 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.066348 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.066470 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.066453 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.066527 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.066489 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k97td\" (UniqueName: \"kubernetes.io/projected/6d871b22-0655-4cc4-9afb-686f9429693f-kube-api-access-k97td\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.168019 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.167980 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.168019 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.168019 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k97td\" (UniqueName: \"kubernetes.io/projected/6d871b22-0655-4cc4-9afb-686f9429693f-kube-api-access-k97td\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.168288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.168143 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d871b22-0655-4cc4-9afb-686f9429693f-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.168288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.168189 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.168288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.168217 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.168288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.168271 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.168513 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.168303 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d871b22-0655-4cc4-9afb-686f9429693f-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.168513 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.168364 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d871b22-0655-4cc4-9afb-686f9429693f-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.168513 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.168432 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.168670 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.168601 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.168933 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.168907 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.169040 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.168994 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.169040 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.168995 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.169204 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.169185 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d871b22-0655-4cc4-9afb-686f9429693f-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.170669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.170623 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.171140 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.171114 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d871b22-0655-4cc4-9afb-686f9429693f-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.191159 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.191126 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k97td\" (UniqueName: \"kubernetes.io/projected/6d871b22-0655-4cc4-9afb-686f9429693f-kube-api-access-k97td\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.191315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.191129 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d871b22-0655-4cc4-9afb-686f9429693f-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.269429 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.269398 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:48.632170 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.632142 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx"] Oct 30 12:20:48.632367 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:20:48.632343 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d871b22_0655_4cc4_9afb_686f9429693f.slice/crio-443f9fd8c62eacb59a359229f88ec2ba45ca0085d46560fc168dd66056393a4d WatchSource:0}: Error finding container 443f9fd8c62eacb59a359229f88ec2ba45ca0085d46560fc168dd66056393a4d: Status 404 returned error can't find the container with id 443f9fd8c62eacb59a359229f88ec2ba45ca0085d46560fc168dd66056393a4d Oct 30 12:20:48.635096 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.635064 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:20:48.635165 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.635128 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:20:48.635165 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.635156 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:20:48.858940 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.858906 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" event={"ID":"6d871b22-0655-4cc4-9afb-686f9429693f","Type":"ContainerStarted","Data":"0194586883c3a81b4ec3633f51fca293ab37e9e6ded84c7441425550cbe1d0fe"} Oct 30 12:20:48.859133 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:48.858954 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" event={"ID":"6d871b22-0655-4cc4-9afb-686f9429693f","Type":"ContainerStarted","Data":"443f9fd8c62eacb59a359229f88ec2ba45ca0085d46560fc168dd66056393a4d"} Oct 30 12:20:49.270615 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:49.270581 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:49.272366 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:49.272344 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:20:49.272466 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:49.272393 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" podUID="6d871b22-0655-4cc4-9afb-686f9429693f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 12:20:49.327176 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:49.327141 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:49.823935 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:49.823908 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" Oct 30 12:20:49.866892 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:49.866845 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" podStartSLOduration=2.866830646 podStartE2EDuration="2.866830646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:48.909522476 +0000 UTC m=+1602.019365360" watchObservedRunningTime="2026-10-30 12:20:49.866830646 +0000 UTC m=+1602.976673526" Oct 30 12:20:50.269977 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:50.269941 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:20:50.270170 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:50.270017 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" podUID="6d871b22-0655-4cc4-9afb-686f9429693f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 12:20:51.240972 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:51.240913 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:51.270453 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:51.270421 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:20:51.270613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:51.270478 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" podUID="6d871b22-0655-4cc4-9afb-686f9429693f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 12:20:52.270738 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:52.270703 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:20:52.271171 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:52.270757 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" podUID="6d871b22-0655-4cc4-9afb-686f9429693f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 12:20:53.250301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.250268 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g"] Oct 30 12:20:53.253697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.253681 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:20:53.269870 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.269844 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:20:53.269985 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.269898 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" podUID="6d871b22-0655-4cc4-9afb-686f9429693f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 12:20:53.276465 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.276439 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-82299a-serving-cert\"" Oct 30 12:20:53.323519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.323489 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g"] Oct 30 12:20:53.359871 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.359845 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:53.372385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.372361 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4350a054-e8ff-435a-b68f-3b2a89bda23c-maas-api-tls\") pod \"maas-api-e2e-api-a-82299a-6659bf8dd9-b426g\" (UID: \"4350a054-e8ff-435a-b68f-3b2a89bda23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:20:53.372537 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.372389 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4350a054-e8ff-435a-b68f-3b2a89bda23c-openshift-service-ca\") pod \"maas-api-e2e-api-a-82299a-6659bf8dd9-b426g\" (UID: \"4350a054-e8ff-435a-b68f-3b2a89bda23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:20:53.372537 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.372427 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqq9v\" (UniqueName: \"kubernetes.io/projected/4350a054-e8ff-435a-b68f-3b2a89bda23c-kube-api-access-rqq9v\") pod \"maas-api-e2e-api-a-82299a-6659bf8dd9-b426g\" (UID: \"4350a054-e8ff-435a-b68f-3b2a89bda23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:20:53.372628 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.372589 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4350a054-e8ff-435a-b68f-3b2a89bda23c-metrics-certs\") pod \"maas-api-e2e-api-a-82299a-6659bf8dd9-b426g\" (UID: \"4350a054-e8ff-435a-b68f-3b2a89bda23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:20:53.475386 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.475346 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4350a054-e8ff-435a-b68f-3b2a89bda23c-metrics-certs\") pod \"maas-api-e2e-api-a-82299a-6659bf8dd9-b426g\" (UID: \"4350a054-e8ff-435a-b68f-3b2a89bda23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:20:53.477584 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.475613 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:53.477584 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.476130 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4350a054-e8ff-435a-b68f-3b2a89bda23c-maas-api-tls\") pod \"maas-api-e2e-api-a-82299a-6659bf8dd9-b426g\" (UID: \"4350a054-e8ff-435a-b68f-3b2a89bda23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:20:53.477584 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.476166 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4350a054-e8ff-435a-b68f-3b2a89bda23c-openshift-service-ca\") pod \"maas-api-e2e-api-a-82299a-6659bf8dd9-b426g\" (UID: \"4350a054-e8ff-435a-b68f-3b2a89bda23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:20:53.477584 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.476225 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqq9v\" (UniqueName: \"kubernetes.io/projected/4350a054-e8ff-435a-b68f-3b2a89bda23c-kube-api-access-rqq9v\") pod \"maas-api-e2e-api-a-82299a-6659bf8dd9-b426g\" (UID: \"4350a054-e8ff-435a-b68f-3b2a89bda23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:20:53.477584 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.477023 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4350a054-e8ff-435a-b68f-3b2a89bda23c-openshift-service-ca\") pod \"maas-api-e2e-api-a-82299a-6659bf8dd9-b426g\" (UID: \"4350a054-e8ff-435a-b68f-3b2a89bda23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:20:53.477584 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:20:53.477129 4413 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-serving-cert: secret "maas-api-e2e-api-a-82299a-serving-cert" not found Oct 30 12:20:53.478350 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:20:53.478323 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4350a054-e8ff-435a-b68f-3b2a89bda23c-maas-api-tls podName:4350a054-e8ff-435a-b68f-3b2a89bda23c nodeName:}" failed. No retries permitted until 2026-10-30 12:20:53.978293726 +0000 UTC m=+1607.088136605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4350a054-e8ff-435a-b68f-3b2a89bda23c-maas-api-tls") pod "maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" (UID: "4350a054-e8ff-435a-b68f-3b2a89bda23c") : secret "maas-api-e2e-api-a-82299a-serving-cert" not found Oct 30 12:20:53.479900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.479874 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4350a054-e8ff-435a-b68f-3b2a89bda23c-metrics-certs\") pod \"maas-api-e2e-api-a-82299a-6659bf8dd9-b426g\" (UID: \"4350a054-e8ff-435a-b68f-3b2a89bda23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:20:53.548338 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.548239 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqq9v\" (UniqueName: \"kubernetes.io/projected/4350a054-e8ff-435a-b68f-3b2a89bda23c-kube-api-access-rqq9v\") pod \"maas-api-e2e-api-a-82299a-6659bf8dd9-b426g\" (UID: \"4350a054-e8ff-435a-b68f-3b2a89bda23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:20:53.669621 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.669593 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:53.986378 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.986340 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4350a054-e8ff-435a-b68f-3b2a89bda23c-maas-api-tls\") pod \"maas-api-e2e-api-a-82299a-6659bf8dd9-b426g\" (UID: \"4350a054-e8ff-435a-b68f-3b2a89bda23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:20:53.989386 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:53.989360 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4350a054-e8ff-435a-b68f-3b2a89bda23c-maas-api-tls\") pod \"maas-api-e2e-api-a-82299a-6659bf8dd9-b426g\" (UID: \"4350a054-e8ff-435a-b68f-3b2a89bda23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:20:54.167844 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:54.167813 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:20:54.270343 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:54.270315 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:20:54.270499 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:54.270361 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" podUID="6d871b22-0655-4cc4-9afb-686f9429693f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 12:20:54.517926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:54.517737 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g"] Oct 30 12:20:54.523719 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:20:54.523668 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4350a054_e8ff_435a_b68f_3b2a89bda23c.slice/crio-4b1c9042de21acea33a9f5766bd4804cb28e15d4b9858aab0b6556ada4d31069 WatchSource:0}: Error finding container 4b1c9042de21acea33a9f5766bd4804cb28e15d4b9858aab0b6556ada4d31069: Status 404 returned error can't find the container with id 4b1c9042de21acea33a9f5766bd4804cb28e15d4b9858aab0b6556ada4d31069 Oct 30 12:20:54.928658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:54.928612 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" event={"ID":"4350a054-e8ff-435a-b68f-3b2a89bda23c","Type":"ContainerStarted","Data":"4b1c9042de21acea33a9f5766bd4804cb28e15d4b9858aab0b6556ada4d31069"} Oct 30 12:20:55.270430 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:55.270339 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 12:20:55.270430 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:55.270399 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" podUID="6d871b22-0655-4cc4-9afb-686f9429693f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 12:20:55.761276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:55.758868 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:55.938436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:55.938390 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" event={"ID":"4350a054-e8ff-435a-b68f-3b2a89bda23c","Type":"ContainerStarted","Data":"ecf289e679d42203fcab71dd662a3cb243bfd7a35b17ed15d9eb3015566fa853"} Oct 30 12:20:55.938622 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:55.938540 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:20:55.981761 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:55.981716 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" podStartSLOduration=2.617461754 podStartE2EDuration="2.981699505s" totalImagesPullingTime="364.237751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:55.979760602 +0000 UTC m=+1609.089603491" watchObservedRunningTime="2026-10-30 12:20:55.981699505 +0000 UTC m=+1609.091542386" Oct 30 12:20:56.274651 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:56.274623 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:56.275118 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:56.275095 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:56.275865 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:56.275848 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:20:56.770982 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:56.770945 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:57.558243 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:57.558213 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:20:59.322647 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:20:59.322611 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:00.788257 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:00.788226 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:00.961498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:00.961465 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:01.949973 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:01.949947 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:21:02.733902 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:02.733868 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:04.550849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:04.550813 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:05.814574 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:21:05.814537 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:21:06.488026 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:06.487986 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:08.013402 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:08.013353 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:08.233640 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:08.233607 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:08.359107 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:08.359024 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:10.268147 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:10.268112 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:12.318594 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:12.318555 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:14.083083 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:14.083047 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:15.885697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:15.885633 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:16.110503 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:16.110474 4413 generic.go:356] "Generic (PLEG): container finished" podID="42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" containerID="97b96593d501ed9d9131cdf7055ed4baa97c4a68c19b584b1256cb90725bd513" exitCode=0 Oct 30 12:21:16.110666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:16.110549 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" event={"ID":"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d","Type":"ContainerDied","Data":"97b96593d501ed9d9131cdf7055ed4baa97c4a68c19b584b1256cb90725bd513"} Oct 30 12:21:16.111010 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:16.110997 4413 scope.go:122] "RemoveContainer" containerID="97b96593d501ed9d9131cdf7055ed4baa97c4a68c19b584b1256cb90725bd513" Oct 30 12:21:16.112676 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:16.112646 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:21:16.112786 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:16.112718 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:21:16.112786 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:16.112749 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:21:17.127217 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:17.127141 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" event={"ID":"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d","Type":"ContainerStarted","Data":"cbb155d60daf1bdc1959aa3db9a7cc0d158dd3eeb3030a8acf845740d0d4c5f3"} Oct 30 12:21:17.520777 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:17.520687 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:17.640563 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:17.640527 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:17.957167 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:17.957135 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:21:17.958621 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:17.958590 4413 patch_prober.go:29] interesting pod/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 12:21:17.958737 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:17.958651 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" podUID="42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 12:21:18.317690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:18.317597 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:18.768520 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:18.768474 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:18.958749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:18.958704 4413 patch_prober.go:29] interesting pod/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 12:21:18.959314 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:18.958781 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" podUID="42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 12:21:19.160780 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:19.160744 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:19.677586 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:19.677555 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:19.958297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:19.958193 4413 patch_prober.go:29] interesting pod/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 12:21:19.958297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:19.958278 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" podUID="42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 12:21:20.734116 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:20.734077 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:21:20.957700 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:20.957647 4413 patch_prober.go:29] interesting pod/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 12:21:20.957862 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:20.957729 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" podUID="42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 12:21:21.453936 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:21.453893 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:21.957363 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:21.957324 4413 patch_prober.go:29] interesting pod/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 12:21:21.957831 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:21.957379 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" podUID="42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 12:21:22.171330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:22.170625 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:22.284321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:22.284199 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:22.962222 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:22.962186 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:21:22.963617 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:22.963581 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:21:22.990686 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:22.990657 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:23.150449 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:23.150362 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:23.953144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:23.953112 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:24.749014 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:24.748983 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:24.860927 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:24.860898 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:25.178908 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:25.178872 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk"] Oct 30 12:21:25.194477 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:25.194442 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk"] Oct 30 12:21:25.197372 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:25.197342 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk" Oct 30 12:21:25.224364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:25.224327 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ebc6bb\"" Oct 30 12:21:25.224530 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:25.224510 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ebc6bb-dockercfg-jpfmp\"" Oct 30 12:21:25.284861 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:25.284826 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mwpr\" (UniqueName: \"kubernetes.io/projected/b401ad16-659d-472c-9934-26785d401eff-kube-api-access-9mwpr\") pod \"payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk\" (UID: \"b401ad16-659d-472c-9934-26785d401eff\") " pod="openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk" Oct 30 12:21:25.285102 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:25.285078 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b401ad16-659d-472c-9934-26785d401eff-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk\" (UID: \"b401ad16-659d-472c-9934-26785d401eff\") " pod="openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk" Oct 30 12:21:25.386714 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:25.386682 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b401ad16-659d-472c-9934-26785d401eff-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk\" (UID: \"b401ad16-659d-472c-9934-26785d401eff\") " pod="openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk" Oct 30 12:21:25.386915 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:25.386824 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mwpr\" (UniqueName: \"kubernetes.io/projected/b401ad16-659d-472c-9934-26785d401eff-kube-api-access-9mwpr\") pod \"payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk\" (UID: \"b401ad16-659d-472c-9934-26785d401eff\") " pod="openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk" Oct 30 12:21:25.387732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:25.387703 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b401ad16-659d-472c-9934-26785d401eff-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk\" (UID: \"b401ad16-659d-472c-9934-26785d401eff\") " pod="openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk" Oct 30 12:21:25.410197 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:25.410164 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mwpr\" (UniqueName: \"kubernetes.io/projected/b401ad16-659d-472c-9934-26785d401eff-kube-api-access-9mwpr\") pod \"payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk\" (UID: \"b401ad16-659d-472c-9934-26785d401eff\") " pod="openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk" Oct 30 12:21:25.519404 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:25.519305 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk" Oct 30 12:21:25.821304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:25.821268 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk"] Oct 30 12:21:25.822120 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:21:25.822093 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb401ad16_659d_472c_9934_26785d401eff.slice/crio-1109b6dfd59f8907a9ee1a6ff13cbe061377af5cbf081df8972e92dcc8524c89 WatchSource:0}: Error finding container 1109b6dfd59f8907a9ee1a6ff13cbe061377af5cbf081df8972e92dcc8524c89: Status 404 returned error can't find the container with id 1109b6dfd59f8907a9ee1a6ff13cbe061377af5cbf081df8972e92dcc8524c89 Oct 30 12:21:25.863409 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:25.863367 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:26.208622 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:26.208592 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk" event={"ID":"b401ad16-659d-472c-9934-26785d401eff","Type":"ContainerStarted","Data":"525edd9edc00f361a0a0a9218040f5c8d30bc3db85f6f01abe5adf08af09405b"} Oct 30 12:21:26.208810 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:26.208642 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk" event={"ID":"b401ad16-659d-472c-9934-26785d401eff","Type":"ContainerStarted","Data":"1109b6dfd59f8907a9ee1a6ff13cbe061377af5cbf081df8972e92dcc8524c89"} Oct 30 12:21:26.208810 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:26.208787 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk" Oct 30 12:21:26.249658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:26.249601 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk" podStartSLOduration=2.249585136 podStartE2EDuration="2.249585136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:21:26.247460522 +0000 UTC m=+1639.357303403" watchObservedRunningTime="2026-10-30 12:21:26.249585136 +0000 UTC m=+1639.359428032" Oct 30 12:21:27.835766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:27.835735 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:29.555739 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:29.555705 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:31.152024 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:31.151955 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:32.857913 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:32.857876 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:34.451028 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:34.450995 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:35.833626 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:21:35.833582 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:21:36.420171 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:36.420126 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:36.600430 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:36.600397 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m"] Oct 30 12:21:36.829425 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:36.829222 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:36.966995 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:36.966962 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:37.261277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:37.260483 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk" Oct 30 12:21:38.856768 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:38.856736 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:40.599178 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:40.599143 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:40.718011 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:40.717968 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:41.600894 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:41.600841 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" podUID="db4d3608-4ffc-4a96-9d5d-48d8de49facf" containerName="payload-processing" containerID="cri-o://32d28dafc690c0f100a7e8c2b85dba4bb0a0fb3e735d942413e6d17203e10e00" gracePeriod=25 Oct 30 12:21:42.366451 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:42.366417 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:43.835723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:43.835685 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:44.383088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:44.383043 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:44.571051 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:44.571015 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:44.703965 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:44.703885 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:45.479046 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:45.479011 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:46.357689 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:46.357651 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:47.964396 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:47.964365 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:49.853019 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:49.852940 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:50.277091 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:50.277055 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" podUID="db4d3608-4ffc-4a96-9d5d-48d8de49facf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:21:51.488977 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:51.488949 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:51.602694 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:51.602655 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:51.734748 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:51.734711 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:51.926486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:51.926456 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp"] Oct 30 12:21:51.926727 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:51.926704 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" podUID="a3db80a9-3f95-4c9e-a252-78eca98bac38" containerName="istio-proxy" containerID="cri-o://27a457f1228bcd4388bea456c05fd502acace256d3c33b6bfe9d545293c15cab" gracePeriod=30 Oct 30 12:21:51.969185 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:51.969104 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:53.463920 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:53.463877 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:55.251046 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:55.251010 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:55.622233 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:55.622136 4413 patch_prober.go:29] interesting pod/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:21:55.622233 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:55.622212 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" podUID="a3db80a9-3f95-4c9e-a252-78eca98bac38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:21:56.897679 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:56.897636 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:57.040810 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.040764 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:57.395468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.395446 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:21:57.520750 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.520718 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-podinfo\") pod \"a3db80a9-3f95-4c9e-a252-78eca98bac38\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " Oct 30 12:21:57.520750 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.520757 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-token\" (UniqueName: \"kubernetes.io/projected/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-token\") pod \"a3db80a9-3f95-4c9e-a252-78eca98bac38\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " Oct 30 12:21:57.521026 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.520810 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-envoy\") pod \"a3db80a9-3f95-4c9e-a252-78eca98bac38\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " Oct 30 12:21:57.521026 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.520893 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-workload-socket\") pod \"a3db80a9-3f95-4c9e-a252-78eca98bac38\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " Oct 30 12:21:57.521026 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.520956 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3db80a9-3f95-4c9e-a252-78eca98bac38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3db80a9-3f95-4c9e-a252-78eca98bac38-istiod-ca-cert\") pod \"a3db80a9-3f95-4c9e-a252-78eca98bac38\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " Oct 30 12:21:57.521026 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.520977 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-credential-socket\") pod \"a3db80a9-3f95-4c9e-a252-78eca98bac38\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " Oct 30 12:21:57.521267 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.521044 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-data\") pod \"a3db80a9-3f95-4c9e-a252-78eca98bac38\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " Oct 30 12:21:57.521267 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.521110 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-workload-certs\") pod \"a3db80a9-3f95-4c9e-a252-78eca98bac38\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " Oct 30 12:21:57.521267 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.521178 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3db80a9-3f95-4c9e-a252-78eca98bac38-kube-api-access-q6pw4\" (UniqueName: \"kubernetes.io/projected/a3db80a9-3f95-4c9e-a252-78eca98bac38-kube-api-access-q6pw4\") pod \"a3db80a9-3f95-4c9e-a252-78eca98bac38\" (UID: \"a3db80a9-3f95-4c9e-a252-78eca98bac38\") " Oct 30 12:21:57.521267 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.521222 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-workload-socket" pod "a3db80a9-3f95-4c9e-a252-78eca98bac38" (UID: "a3db80a9-3f95-4c9e-a252-78eca98bac38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:21:57.521498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.521425 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3db80a9-3f95-4c9e-a252-78eca98bac38-istiod-ca-cert" pod "a3db80a9-3f95-4c9e-a252-78eca98bac38" (UID: "a3db80a9-3f95-4c9e-a252-78eca98bac38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:21:57.521498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.521446 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-data" pod "a3db80a9-3f95-4c9e-a252-78eca98bac38" (UID: "a3db80a9-3f95-4c9e-a252-78eca98bac38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:21:57.521784 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.521762 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-workload-certs" pod "a3db80a9-3f95-4c9e-a252-78eca98bac38" (UID: "a3db80a9-3f95-4c9e-a252-78eca98bac38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:21:57.522178 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.522153 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-credential-socket" pod "a3db80a9-3f95-4c9e-a252-78eca98bac38" (UID: "a3db80a9-3f95-4c9e-a252-78eca98bac38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:21:57.522317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.522198 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:21:57.522317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.522215 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:21:57.522317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.522229 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:21:57.522317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.522243 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3db80a9-3f95-4c9e-a252-78eca98bac38-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:21:57.523628 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.523484 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-envoy" pod "a3db80a9-3f95-4c9e-a252-78eca98bac38" (UID: "a3db80a9-3f95-4c9e-a252-78eca98bac38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:21:57.523628 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.523586 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-podinfo" pod "a3db80a9-3f95-4c9e-a252-78eca98bac38" (UID: "a3db80a9-3f95-4c9e-a252-78eca98bac38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:21:57.523628 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.523597 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-token" pod "a3db80a9-3f95-4c9e-a252-78eca98bac38" (UID: "a3db80a9-3f95-4c9e-a252-78eca98bac38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:21:57.523839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.523781 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3db80a9-3f95-4c9e-a252-78eca98bac38-kube-api-access-q6pw4" pod "a3db80a9-3f95-4c9e-a252-78eca98bac38" (UID: "a3db80a9-3f95-4c9e-a252-78eca98bac38"). InnerVolumeSpecName "kube-api-access-q6pw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:21:57.523929 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.523908 4413 generic.go:356] "Generic (PLEG): container finished" podID="a3db80a9-3f95-4c9e-a252-78eca98bac38" containerID="27a457f1228bcd4388bea456c05fd502acace256d3c33b6bfe9d545293c15cab" exitCode=0 Oct 30 12:21:57.523993 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.523977 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" Oct 30 12:21:57.524031 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.523985 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" event={"ID":"a3db80a9-3f95-4c9e-a252-78eca98bac38","Type":"ContainerDied","Data":"27a457f1228bcd4388bea456c05fd502acace256d3c33b6bfe9d545293c15cab"} Oct 30 12:21:57.524071 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.524036 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp" event={"ID":"a3db80a9-3f95-4c9e-a252-78eca98bac38","Type":"ContainerDied","Data":"088a6415c7967369d27a3c89721e6a4230d1d4ca17ead9e7454932a2830530b4"} Oct 30 12:21:57.524071 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.524059 4413 scope.go:122] "RemoveContainer" containerID="27a457f1228bcd4388bea456c05fd502acace256d3c33b6bfe9d545293c15cab" Oct 30 12:21:57.558116 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.558093 4413 scope.go:122] "RemoveContainer" containerID="85c68c44a16bdce4eb58df2e99a37811d361bb169154881bf670786023027664" Oct 30 12:21:57.568028 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.568008 4413 scope.go:122] "RemoveContainer" containerID="27a457f1228bcd4388bea456c05fd502acace256d3c33b6bfe9d545293c15cab" Oct 30 12:21:57.568367 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:21:57.568345 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a457f1228bcd4388bea456c05fd502acace256d3c33b6bfe9d545293c15cab\": container with ID starting with 27a457f1228bcd4388bea456c05fd502acace256d3c33b6bfe9d545293c15cab not found: ID does not exist" containerID="27a457f1228bcd4388bea456c05fd502acace256d3c33b6bfe9d545293c15cab" Oct 30 12:21:57.568450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.568379 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a457f1228bcd4388bea456c05fd502acace256d3c33b6bfe9d545293c15cab"} err="failed to get container status \"27a457f1228bcd4388bea456c05fd502acace256d3c33b6bfe9d545293c15cab\": rpc error: code = NotFound desc = could not find container \"27a457f1228bcd4388bea456c05fd502acace256d3c33b6bfe9d545293c15cab\": container with ID starting with 27a457f1228bcd4388bea456c05fd502acace256d3c33b6bfe9d545293c15cab not found: ID does not exist" Oct 30 12:21:57.568450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.568403 4413 scope.go:122] "RemoveContainer" containerID="85c68c44a16bdce4eb58df2e99a37811d361bb169154881bf670786023027664" Oct 30 12:21:57.568698 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:21:57.568672 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85c68c44a16bdce4eb58df2e99a37811d361bb169154881bf670786023027664\": container with ID starting with 85c68c44a16bdce4eb58df2e99a37811d361bb169154881bf670786023027664 not found: ID does not exist" containerID="85c68c44a16bdce4eb58df2e99a37811d361bb169154881bf670786023027664" Oct 30 12:21:57.568778 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.568711 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85c68c44a16bdce4eb58df2e99a37811d361bb169154881bf670786023027664"} err="failed to get container status \"85c68c44a16bdce4eb58df2e99a37811d361bb169154881bf670786023027664\": rpc error: code = NotFound desc = could not find container \"85c68c44a16bdce4eb58df2e99a37811d361bb169154881bf670786023027664\": container with ID starting with 85c68c44a16bdce4eb58df2e99a37811d361bb169154881bf670786023027664 not found: ID does not exist" Oct 30 12:21:57.591951 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.591918 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp"] Oct 30 12:21:57.595206 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.595184 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5649cd-openshift-default-5d487d7975-ssfwp"] Oct 30 12:21:57.623881 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.623862 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:21:57.623981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.623885 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6pw4\" (UniqueName: \"kubernetes.io/projected/a3db80a9-3f95-4c9e-a252-78eca98bac38-kube-api-access-q6pw4\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:21:57.623981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.623896 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:21:57.623981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.623904 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:21:57.623981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:57.623913 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3db80a9-3f95-4c9e-a252-78eca98bac38-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:21:59.020787 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:59.020756 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3db80a9-3f95-4c9e-a252-78eca98bac38" path="/var/lib/kubelet/pods/a3db80a9-3f95-4c9e-a252-78eca98bac38/volumes" Oct 30 12:21:59.040655 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:59.040627 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:59.150450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:59.150414 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:21:59.539656 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:21:59.539629 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:00.098539 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:00.098507 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:00.277741 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:00.277699 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" podUID="db4d3608-4ffc-4a96-9d5d-48d8de49facf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:22:01.049229 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:01.049199 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:01.889418 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:01.889389 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" Oct 30 12:22:01.996474 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:01.996445 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4d3608-4ffc-4a96-9d5d-48d8de49facf-kube-api-access-wwbrd\" (UniqueName: \"kubernetes.io/projected/db4d3608-4ffc-4a96-9d5d-48d8de49facf-kube-api-access-wwbrd\") pod \"db4d3608-4ffc-4a96-9d5d-48d8de49facf\" (UID: \"db4d3608-4ffc-4a96-9d5d-48d8de49facf\") " Oct 30 12:22:01.996688 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:01.996674 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4d3608-4ffc-4a96-9d5d-48d8de49facf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4d3608-4ffc-4a96-9d5d-48d8de49facf-plugins-config-volume\") pod \"db4d3608-4ffc-4a96-9d5d-48d8de49facf\" (UID: \"db4d3608-4ffc-4a96-9d5d-48d8de49facf\") " Oct 30 12:22:01.997022 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:01.996996 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4d3608-4ffc-4a96-9d5d-48d8de49facf-plugins-config-volume" pod "db4d3608-4ffc-4a96-9d5d-48d8de49facf" (UID: "db4d3608-4ffc-4a96-9d5d-48d8de49facf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:22:01.997160 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:01.997110 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4d3608-4ffc-4a96-9d5d-48d8de49facf-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:22:01.999205 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:01.999180 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4d3608-4ffc-4a96-9d5d-48d8de49facf-kube-api-access-wwbrd" pod "db4d3608-4ffc-4a96-9d5d-48d8de49facf" (UID: "db4d3608-4ffc-4a96-9d5d-48d8de49facf"). InnerVolumeSpecName "kube-api-access-wwbrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:22:02.099408 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:02.099372 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwbrd\" (UniqueName: \"kubernetes.io/projected/db4d3608-4ffc-4a96-9d5d-48d8de49facf-kube-api-access-wwbrd\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:22:02.561882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:02.561849 4413 generic.go:356] "Generic (PLEG): container finished" podID="db4d3608-4ffc-4a96-9d5d-48d8de49facf" containerID="32d28dafc690c0f100a7e8c2b85dba4bb0a0fb3e735d942413e6d17203e10e00" exitCode=0 Oct 30 12:22:02.562097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:02.561944 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" event={"ID":"db4d3608-4ffc-4a96-9d5d-48d8de49facf","Type":"ContainerDied","Data":"32d28dafc690c0f100a7e8c2b85dba4bb0a0fb3e735d942413e6d17203e10e00"} Oct 30 12:22:02.562097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:02.561994 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" event={"ID":"db4d3608-4ffc-4a96-9d5d-48d8de49facf","Type":"ContainerDied","Data":"7a60b549831c16c544a1c90ba1bc28c59e9a29f65dd3894037e05d8bed399d21"} Oct 30 12:22:02.562097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:02.561947 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m" Oct 30 12:22:02.562097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:02.562022 4413 scope.go:122] "RemoveContainer" containerID="32d28dafc690c0f100a7e8c2b85dba4bb0a0fb3e735d942413e6d17203e10e00" Oct 30 12:22:02.573675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:02.573648 4413 scope.go:122] "RemoveContainer" containerID="32d28dafc690c0f100a7e8c2b85dba4bb0a0fb3e735d942413e6d17203e10e00" Oct 30 12:22:02.573981 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:22:02.573963 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d28dafc690c0f100a7e8c2b85dba4bb0a0fb3e735d942413e6d17203e10e00\": container with ID starting with 32d28dafc690c0f100a7e8c2b85dba4bb0a0fb3e735d942413e6d17203e10e00 not found: ID does not exist" containerID="32d28dafc690c0f100a7e8c2b85dba4bb0a0fb3e735d942413e6d17203e10e00" Oct 30 12:22:02.574060 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:02.573993 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d28dafc690c0f100a7e8c2b85dba4bb0a0fb3e735d942413e6d17203e10e00"} err="failed to get container status \"32d28dafc690c0f100a7e8c2b85dba4bb0a0fb3e735d942413e6d17203e10e00\": rpc error: code = NotFound desc = could not find container \"32d28dafc690c0f100a7e8c2b85dba4bb0a0fb3e735d942413e6d17203e10e00\": container with ID starting with 32d28dafc690c0f100a7e8c2b85dba4bb0a0fb3e735d942413e6d17203e10e00 not found: ID does not exist" Oct 30 12:22:02.614170 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:02.614134 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m"] Oct 30 12:22:02.626121 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:02.626071 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-060f41f2-85b6c7b85d-mz52m"] Oct 30 12:22:02.768827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:02.768795 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:03.021826 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:03.021785 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4d3608-4ffc-4a96-9d5d-48d8de49facf" path="/var/lib/kubelet/pods/db4d3608-4ffc-4a96-9d5d-48d8de49facf/volumes" Oct 30 12:22:03.032377 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:03.032345 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll"] Oct 30 12:22:03.145000 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:03.144956 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:03.272893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:03.272805 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:03.469776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:03.469726 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:03.708753 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:03.708716 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:04.896100 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:04.896070 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:05.009982 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.009948 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:05.065281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.065235 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw"] Oct 30 12:22:05.065957 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.065943 4413 cpu_manager.go:414] "containerMap: removing container" podUID="a3db80a9-3f95-4c9e-a252-78eca98bac38" containerName="istio-proxy" Oct 30 12:22:05.066002 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.065959 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3db80a9-3f95-4c9e-a252-78eca98bac38" containerName="istio-proxy" Oct 30 12:22:05.066002 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.065977 4413 cpu_manager.go:414] "containerMap: removing container" podUID="db4d3608-4ffc-4a96-9d5d-48d8de49facf" containerName="payload-processing" Oct 30 12:22:05.066002 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.065982 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4d3608-4ffc-4a96-9d5d-48d8de49facf" containerName="payload-processing" Oct 30 12:22:05.066101 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.066075 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3db80a9-3f95-4c9e-a252-78eca98bac38" containerName="istio-proxy" Oct 30 12:22:05.066101 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.066085 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3db80a9-3f95-4c9e-a252-78eca98bac38" containerName="istio-proxy" Oct 30 12:22:05.066101 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.066091 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4d3608-4ffc-4a96-9d5d-48d8de49facf" containerName="payload-processing" Oct 30 12:22:05.069781 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.069765 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw" Oct 30 12:22:05.087794 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.087770 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw"] Oct 30 12:22:05.095534 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.095507 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c35d6b\"" Oct 30 12:22:05.095694 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.095506 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c35d6b-dockercfg-fkdtx\"" Oct 30 12:22:05.158287 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.158184 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064df158-8c5e-4cab-bef7-9b9e2a64f460-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw\" (UID: \"064df158-8c5e-4cab-bef7-9b9e2a64f460\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw" Oct 30 12:22:05.160394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.160370 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9n89\" (UniqueName: \"kubernetes.io/projected/064df158-8c5e-4cab-bef7-9b9e2a64f460-kube-api-access-z9n89\") pod \"payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw\" (UID: \"064df158-8c5e-4cab-bef7-9b9e2a64f460\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw" Oct 30 12:22:05.172197 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.172161 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx"] Oct 30 12:22:05.173424 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.173399 4413 cpu_manager.go:414] "containerMap: removing container" podUID="a3db80a9-3f95-4c9e-a252-78eca98bac38" containerName="istio-proxy" Oct 30 12:22:05.173585 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.173569 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3db80a9-3f95-4c9e-a252-78eca98bac38" containerName="istio-proxy" Oct 30 12:22:05.307698 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.305943 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9n89\" (UniqueName: \"kubernetes.io/projected/064df158-8c5e-4cab-bef7-9b9e2a64f460-kube-api-access-z9n89\") pod \"payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw\" (UID: \"064df158-8c5e-4cab-bef7-9b9e2a64f460\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw" Oct 30 12:22:05.311548 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.310313 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:05.311548 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.310614 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064df158-8c5e-4cab-bef7-9b9e2a64f460-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw\" (UID: \"064df158-8c5e-4cab-bef7-9b9e2a64f460\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw" Oct 30 12:22:05.327280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.323641 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064df158-8c5e-4cab-bef7-9b9e2a64f460-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw\" (UID: \"064df158-8c5e-4cab-bef7-9b9e2a64f460\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw" Oct 30 12:22:05.343128 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.343087 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx"] Oct 30 12:22:05.343326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.343282 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx" Oct 30 12:22:05.370983 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.370954 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9n89\" (UniqueName: \"kubernetes.io/projected/064df158-8c5e-4cab-bef7-9b9e2a64f460-kube-api-access-z9n89\") pod \"payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw\" (UID: \"064df158-8c5e-4cab-bef7-9b9e2a64f460\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw" Oct 30 12:22:05.383668 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.383633 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw" Oct 30 12:22:05.439879 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.439793 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d6d475-531b-435f-b15b-65e7ed8437b4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c35d6b-74886576d-96flx\" (UID: \"68d6d475-531b-435f-b15b-65e7ed8437b4\") " pod="openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx" Oct 30 12:22:05.440021 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.439961 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk6k6\" (UniqueName: \"kubernetes.io/projected/68d6d475-531b-435f-b15b-65e7ed8437b4-kube-api-access-sk6k6\") pod \"payload-processing-e2e-api-a-c35d6b-74886576d-96flx\" (UID: \"68d6d475-531b-435f-b15b-65e7ed8437b4\") " pod="openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx" Oct 30 12:22:05.542600 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.542091 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk6k6\" (UniqueName: \"kubernetes.io/projected/68d6d475-531b-435f-b15b-65e7ed8437b4-kube-api-access-sk6k6\") pod \"payload-processing-e2e-api-a-c35d6b-74886576d-96flx\" (UID: \"68d6d475-531b-435f-b15b-65e7ed8437b4\") " pod="openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx" Oct 30 12:22:05.542600 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.542449 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d6d475-531b-435f-b15b-65e7ed8437b4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c35d6b-74886576d-96flx\" (UID: \"68d6d475-531b-435f-b15b-65e7ed8437b4\") " pod="openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx" Oct 30 12:22:05.543495 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.543471 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d6d475-531b-435f-b15b-65e7ed8437b4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c35d6b-74886576d-96flx\" (UID: \"68d6d475-531b-435f-b15b-65e7ed8437b4\") " pod="openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx" Oct 30 12:22:05.567082 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.566971 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk6k6\" (UniqueName: \"kubernetes.io/projected/68d6d475-531b-435f-b15b-65e7ed8437b4-kube-api-access-sk6k6\") pod \"payload-processing-e2e-api-a-c35d6b-74886576d-96flx\" (UID: \"68d6d475-531b-435f-b15b-65e7ed8437b4\") " pod="openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx" Oct 30 12:22:05.672697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.672667 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx" Oct 30 12:22:05.714986 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.701656 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw"] Oct 30 12:22:05.826141 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:22:05.826068 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:22:05.988672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:05.988645 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx"] Oct 30 12:22:05.990085 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:22:05.990057 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d6d475_531b_435f_b15b_65e7ed8437b4.slice/crio-3109dabf76b285f23b5599fa5cd124a73c2634670c7da8525b9738412fada6cb WatchSource:0}: Error finding container 3109dabf76b285f23b5599fa5cd124a73c2634670c7da8525b9738412fada6cb: Status 404 returned error can't find the container with id 3109dabf76b285f23b5599fa5cd124a73c2634670c7da8525b9738412fada6cb Oct 30 12:22:06.598264 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:06.598206 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx" event={"ID":"68d6d475-531b-435f-b15b-65e7ed8437b4","Type":"ContainerStarted","Data":"4cc4b274e933585e28e753e0118d48114da528bd78c991d03037f76d90abe77f"} Oct 30 12:22:06.598478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:06.598328 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx" event={"ID":"68d6d475-531b-435f-b15b-65e7ed8437b4","Type":"ContainerStarted","Data":"3109dabf76b285f23b5599fa5cd124a73c2634670c7da8525b9738412fada6cb"} Oct 30 12:22:06.598478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:06.598361 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx" Oct 30 12:22:06.600148 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:06.600109 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw" event={"ID":"064df158-8c5e-4cab-bef7-9b9e2a64f460","Type":"ContainerStarted","Data":"a3953db329e7e9f2493eaeec5557c3d8dba577a12c24914c5332ef6ece4fbe52"} Oct 30 12:22:06.600321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:06.600160 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw" event={"ID":"064df158-8c5e-4cab-bef7-9b9e2a64f460","Type":"ContainerStarted","Data":"5a9fb170ec334be5733c4b1e04a42e9cf65f7a1756c4195dab1c7a227357e693"} Oct 30 12:22:06.600321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:06.600198 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw" Oct 30 12:22:06.641590 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:06.641531 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx" podStartSLOduration=1.641516047 podStartE2EDuration="1.641516047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:06.637476504 +0000 UTC m=+1679.747319386" watchObservedRunningTime="2026-10-30 12:22:06.641516047 +0000 UTC m=+1679.751358927" Oct 30 12:22:06.679481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:06.679424 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw" podStartSLOduration=1.679410107 podStartE2EDuration="1.679410107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:06.675437633 +0000 UTC m=+1679.785280516" watchObservedRunningTime="2026-10-30 12:22:06.679410107 +0000 UTC m=+1679.789252987" Oct 30 12:22:06.887827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:06.887790 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:07.639713 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:07.639667 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:07.750700 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:07.750665 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:07.862235 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:07.862204 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:08.574172 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:08.574125 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll" podUID="f5265e74-61ce-4747-a4b0-df93e28a83cc" containerName="payload-pre-processing" containerID="cri-o://f06544cae3910e4647766a45b6513e45c8ae7f59e8a52626d097203042af3c34" gracePeriod=25 Oct 30 12:22:08.860720 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:08.860697 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll" Oct 30 12:22:08.916543 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:08.916507 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5265e74-61ce-4747-a4b0-df93e28a83cc-kube-api-access-n6vlm\" (UniqueName: \"kubernetes.io/projected/f5265e74-61ce-4747-a4b0-df93e28a83cc-kube-api-access-n6vlm\") pod \"f5265e74-61ce-4747-a4b0-df93e28a83cc\" (UID: \"f5265e74-61ce-4747-a4b0-df93e28a83cc\") " Oct 30 12:22:08.916821 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:08.916799 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5265e74-61ce-4747-a4b0-df93e28a83cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5265e74-61ce-4747-a4b0-df93e28a83cc-plugins-config-volume\") pod \"f5265e74-61ce-4747-a4b0-df93e28a83cc\" (UID: \"f5265e74-61ce-4747-a4b0-df93e28a83cc\") " Oct 30 12:22:08.917298 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:08.917223 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5265e74-61ce-4747-a4b0-df93e28a83cc-plugins-config-volume" pod "f5265e74-61ce-4747-a4b0-df93e28a83cc" (UID: "f5265e74-61ce-4747-a4b0-df93e28a83cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:22:08.917554 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:08.917535 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5265e74-61ce-4747-a4b0-df93e28a83cc-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:22:08.919436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:08.919404 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5265e74-61ce-4747-a4b0-df93e28a83cc-kube-api-access-n6vlm" pod "f5265e74-61ce-4747-a4b0-df93e28a83cc" (UID: "f5265e74-61ce-4747-a4b0-df93e28a83cc"). InnerVolumeSpecName "kube-api-access-n6vlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:22:09.018900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:09.018861 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6vlm\" (UniqueName: \"kubernetes.io/projected/f5265e74-61ce-4747-a4b0-df93e28a83cc-kube-api-access-n6vlm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:22:09.624198 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:09.624161 4413 generic.go:356] "Generic (PLEG): container finished" podID="f5265e74-61ce-4747-a4b0-df93e28a83cc" containerID="f06544cae3910e4647766a45b6513e45c8ae7f59e8a52626d097203042af3c34" exitCode=0 Oct 30 12:22:09.624427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:09.624226 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll" Oct 30 12:22:09.624427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:09.624262 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll" event={"ID":"f5265e74-61ce-4747-a4b0-df93e28a83cc","Type":"ContainerDied","Data":"f06544cae3910e4647766a45b6513e45c8ae7f59e8a52626d097203042af3c34"} Oct 30 12:22:09.624427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:09.624322 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll" event={"ID":"f5265e74-61ce-4747-a4b0-df93e28a83cc","Type":"ContainerDied","Data":"2441878cc8c478edf2d44760efbd8372d849123b08517188508fd6d3cd8227a4"} Oct 30 12:22:09.624427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:09.624348 4413 scope.go:122] "RemoveContainer" containerID="f06544cae3910e4647766a45b6513e45c8ae7f59e8a52626d097203042af3c34" Oct 30 12:22:09.634543 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:09.634521 4413 scope.go:122] "RemoveContainer" containerID="f06544cae3910e4647766a45b6513e45c8ae7f59e8a52626d097203042af3c34" Oct 30 12:22:09.634886 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:22:09.634864 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f06544cae3910e4647766a45b6513e45c8ae7f59e8a52626d097203042af3c34\": container with ID starting with f06544cae3910e4647766a45b6513e45c8ae7f59e8a52626d097203042af3c34 not found: ID does not exist" containerID="f06544cae3910e4647766a45b6513e45c8ae7f59e8a52626d097203042af3c34" Oct 30 12:22:09.634973 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:09.634901 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f06544cae3910e4647766a45b6513e45c8ae7f59e8a52626d097203042af3c34"} err="failed to get container status \"f06544cae3910e4647766a45b6513e45c8ae7f59e8a52626d097203042af3c34\": rpc error: code = NotFound desc = could not find container \"f06544cae3910e4647766a45b6513e45c8ae7f59e8a52626d097203042af3c34\": container with ID starting with f06544cae3910e4647766a45b6513e45c8ae7f59e8a52626d097203042af3c34 not found: ID does not exist" Oct 30 12:22:09.675543 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:09.675508 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll"] Oct 30 12:22:09.687508 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:09.687477 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7566f5-7fb946d9f-5h2ll"] Oct 30 12:22:09.751463 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:09.751430 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:11.026217 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:11.026188 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5265e74-61ce-4747-a4b0-df93e28a83cc" path="/var/lib/kubelet/pods/f5265e74-61ce-4747-a4b0-df93e28a83cc/volumes" Oct 30 12:22:11.454129 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:11.454097 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:12.038233 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:12.038200 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:13.320893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:13.320861 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:15.378274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:15.377854 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:15.941273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:15.940780 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:16.140284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:16.133093 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:17.566658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:17.566628 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:17.613892 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:17.613862 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx" Oct 30 12:22:17.614196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:17.614172 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw" Oct 30 12:22:19.245723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:19.245675 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:19.367492 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:19.367456 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:19.481849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:19.481818 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:20.256545 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:20.256512 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:20.375370 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:20.375325 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:21.342278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:21.342231 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:23.033050 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.033024 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg"] Oct 30 12:22:23.033735 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.033719 4413 cpu_manager.go:414] "containerMap: removing container" podUID="f5265e74-61ce-4747-a4b0-df93e28a83cc" containerName="payload-pre-processing" Oct 30 12:22:23.033784 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.033736 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5265e74-61ce-4747-a4b0-df93e28a83cc" containerName="payload-pre-processing" Oct 30 12:22:23.033864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.033855 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5265e74-61ce-4747-a4b0-df93e28a83cc" containerName="payload-pre-processing" Oct 30 12:22:23.037420 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.037404 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:22:23.047006 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.046985 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-maas-api-tls\") pod \"maas-api-e2e-mt-060f41f2-744745b898-zg6cg\" (UID: \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:22:23.047088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.047032 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-metrics-certs\") pod \"maas-api-e2e-mt-060f41f2-744745b898-zg6cg\" (UID: \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:22:23.047202 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.047182 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-openshift-service-ca\") pod \"maas-api-e2e-mt-060f41f2-744745b898-zg6cg\" (UID: \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:22:23.047325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.047308 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8cmv\" (UniqueName: \"kubernetes.io/projected/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-kube-api-access-l8cmv\") pod \"maas-api-e2e-mt-060f41f2-744745b898-zg6cg\" (UID: \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:22:23.066432 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.066405 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-060f41f2-serving-cert\"" Oct 30 12:22:23.071759 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.071739 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg"] Oct 30 12:22:23.122607 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.122575 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:23.148593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.148521 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-maas-api-tls\") pod \"maas-api-e2e-mt-060f41f2-744745b898-zg6cg\" (UID: \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:22:23.148593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.148582 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-metrics-certs\") pod \"maas-api-e2e-mt-060f41f2-744745b898-zg6cg\" (UID: \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:22:23.148821 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:22:23.148679 4413 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-serving-cert: secret "maas-api-e2e-mt-060f41f2-serving-cert" not found Oct 30 12:22:23.148821 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.148747 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-openshift-service-ca\") pod \"maas-api-e2e-mt-060f41f2-744745b898-zg6cg\" (UID: \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:22:23.148821 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:22:23.148770 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-maas-api-tls podName:cb6b5578-4eac-409a-a8fb-f174ad8aeeb2 nodeName:}" failed. No retries permitted until 2026-10-30 12:22:23.648748858 +0000 UTC m=+1696.758591735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-maas-api-tls") pod "maas-api-e2e-mt-060f41f2-744745b898-zg6cg" (UID: "cb6b5578-4eac-409a-a8fb-f174ad8aeeb2") : secret "maas-api-e2e-mt-060f41f2-serving-cert" not found Oct 30 12:22:23.148989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.148847 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8cmv\" (UniqueName: \"kubernetes.io/projected/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-kube-api-access-l8cmv\") pod \"maas-api-e2e-mt-060f41f2-744745b898-zg6cg\" (UID: \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:22:23.149427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.149410 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-openshift-service-ca\") pod \"maas-api-e2e-mt-060f41f2-744745b898-zg6cg\" (UID: \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:22:23.151380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.151359 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-metrics-certs\") pod \"maas-api-e2e-mt-060f41f2-744745b898-zg6cg\" (UID: \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:22:23.173984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.173961 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8cmv\" (UniqueName: \"kubernetes.io/projected/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-kube-api-access-l8cmv\") pod \"maas-api-e2e-mt-060f41f2-744745b898-zg6cg\" (UID: \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:22:23.238317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.238243 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:23.351453 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.351418 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:23.655666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.655628 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-maas-api-tls\") pod \"maas-api-e2e-mt-060f41f2-744745b898-zg6cg\" (UID: \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:22:23.658580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.658545 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-maas-api-tls\") pod \"maas-api-e2e-mt-060f41f2-744745b898-zg6cg\" (UID: \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:22:23.950035 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:23.949945 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:22:24.260358 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:24.258367 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg"] Oct 30 12:22:24.264161 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:22:24.263896 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6b5578_4eac_409a_a8fb_f174ad8aeeb2.slice/crio-5afa094eabb2156a655a4643cdf3a87f14ad6e3644a524d254cc1aa75fe4b71d WatchSource:0}: Error finding container 5afa094eabb2156a655a4643cdf3a87f14ad6e3644a524d254cc1aa75fe4b71d: Status 404 returned error can't find the container with id 5afa094eabb2156a655a4643cdf3a87f14ad6e3644a524d254cc1aa75fe4b71d Oct 30 12:22:24.764450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:24.764409 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" event={"ID":"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2","Type":"ContainerStarted","Data":"5afa094eabb2156a655a4643cdf3a87f14ad6e3644a524d254cc1aa75fe4b71d"} Oct 30 12:22:24.994736 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:24.994701 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:25.774051 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:25.774016 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" event={"ID":"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2","Type":"ContainerStarted","Data":"48d04cef3fa97257a7e5ab674c40c93b2622b3462ebc7b61bde74063eb8653d7"} Oct 30 12:22:25.774489 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:25.774153 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:22:25.816791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:25.816737 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" podStartSLOduration=2.22860991 podStartE2EDuration="2.81672299s" totalImagesPullingTime="588.11308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:25.815300619 +0000 UTC m=+1698.925143519" watchObservedRunningTime="2026-10-30 12:22:25.81672299 +0000 UTC m=+1698.926565872" Oct 30 12:22:26.661713 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:26.661683 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:28.344939 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:28.344891 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:30.043427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:30.043380 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:30.957414 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:30.957385 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:31.083629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:31.083597 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:31.798633 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:31.798606 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:22:31.853493 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:31.853463 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:33.198759 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.198724 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb"] Oct 30 12:22:33.199281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.199032 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" podUID="50dada57-ddc4-4a37-bdd5-638e44f15a63" containerName="maas-api" containerID="cri-o://1c53e35dda16cea704270a009ae7fcfb50c40c5a6300f3143e666232146dcaea" gracePeriod=30 Oct 30 12:22:33.245656 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.245624 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:33.474376 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.474348 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" Oct 30 12:22:33.620055 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.620025 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50dada57-ddc4-4a37-bdd5-638e44f15a63-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dada57-ddc4-4a37-bdd5-638e44f15a63-metrics-certs\") pod \"50dada57-ddc4-4a37-bdd5-638e44f15a63\" (UID: \"50dada57-ddc4-4a37-bdd5-638e44f15a63\") " Oct 30 12:22:33.620238 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.620170 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50dada57-ddc4-4a37-bdd5-638e44f15a63-kube-api-access-sfxjk\" (UniqueName: \"kubernetes.io/projected/50dada57-ddc4-4a37-bdd5-638e44f15a63-kube-api-access-sfxjk\") pod \"50dada57-ddc4-4a37-bdd5-638e44f15a63\" (UID: \"50dada57-ddc4-4a37-bdd5-638e44f15a63\") " Oct 30 12:22:33.620238 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.620218 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50dada57-ddc4-4a37-bdd5-638e44f15a63-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50dada57-ddc4-4a37-bdd5-638e44f15a63-openshift-service-ca\") pod \"50dada57-ddc4-4a37-bdd5-638e44f15a63\" (UID: \"50dada57-ddc4-4a37-bdd5-638e44f15a63\") " Oct 30 12:22:33.620410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.620290 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50dada57-ddc4-4a37-bdd5-638e44f15a63-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50dada57-ddc4-4a37-bdd5-638e44f15a63-maas-api-tls\") pod \"50dada57-ddc4-4a37-bdd5-638e44f15a63\" (UID: \"50dada57-ddc4-4a37-bdd5-638e44f15a63\") " Oct 30 12:22:33.620662 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.620624 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50dada57-ddc4-4a37-bdd5-638e44f15a63-openshift-service-ca" pod "50dada57-ddc4-4a37-bdd5-638e44f15a63" (UID: "50dada57-ddc4-4a37-bdd5-638e44f15a63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:22:33.621225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.621203 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50dada57-ddc4-4a37-bdd5-638e44f15a63-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:22:33.622435 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.622414 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50dada57-ddc4-4a37-bdd5-638e44f15a63-maas-api-tls" pod "50dada57-ddc4-4a37-bdd5-638e44f15a63" (UID: "50dada57-ddc4-4a37-bdd5-638e44f15a63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:22:33.622508 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.622430 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50dada57-ddc4-4a37-bdd5-638e44f15a63-metrics-certs" pod "50dada57-ddc4-4a37-bdd5-638e44f15a63" (UID: "50dada57-ddc4-4a37-bdd5-638e44f15a63"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:22:33.622636 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.622615 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50dada57-ddc4-4a37-bdd5-638e44f15a63-kube-api-access-sfxjk" pod "50dada57-ddc4-4a37-bdd5-638e44f15a63" (UID: "50dada57-ddc4-4a37-bdd5-638e44f15a63"). InnerVolumeSpecName "kube-api-access-sfxjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:22:33.723560 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.722693 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfxjk\" (UniqueName: \"kubernetes.io/projected/50dada57-ddc4-4a37-bdd5-638e44f15a63-kube-api-access-sfxjk\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:22:33.723560 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.722734 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50dada57-ddc4-4a37-bdd5-638e44f15a63-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:22:33.723560 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.722749 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dada57-ddc4-4a37-bdd5-638e44f15a63-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:22:33.723560 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.723296 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:33.835407 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.835377 4413 generic.go:356] "Generic (PLEG): container finished" podID="50dada57-ddc4-4a37-bdd5-638e44f15a63" containerID="1c53e35dda16cea704270a009ae7fcfb50c40c5a6300f3143e666232146dcaea" exitCode=0 Oct 30 12:22:33.835586 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.835446 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" Oct 30 12:22:33.835586 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.835467 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" event={"ID":"50dada57-ddc4-4a37-bdd5-638e44f15a63","Type":"ContainerDied","Data":"1c53e35dda16cea704270a009ae7fcfb50c40c5a6300f3143e666232146dcaea"} Oct 30 12:22:33.835586 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.835531 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb" event={"ID":"50dada57-ddc4-4a37-bdd5-638e44f15a63","Type":"ContainerDied","Data":"52c357bc441bf59f40c2ebd3132be6bb1b3cb10444bb5f6a71c261d7b9978b3c"} Oct 30 12:22:33.835586 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.835553 4413 scope.go:122] "RemoveContainer" containerID="1c53e35dda16cea704270a009ae7fcfb50c40c5a6300f3143e666232146dcaea" Oct 30 12:22:33.852347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.852322 4413 scope.go:122] "RemoveContainer" containerID="1c53e35dda16cea704270a009ae7fcfb50c40c5a6300f3143e666232146dcaea" Oct 30 12:22:33.852699 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:22:33.852675 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c53e35dda16cea704270a009ae7fcfb50c40c5a6300f3143e666232146dcaea\": container with ID starting with 1c53e35dda16cea704270a009ae7fcfb50c40c5a6300f3143e666232146dcaea not found: ID does not exist" containerID="1c53e35dda16cea704270a009ae7fcfb50c40c5a6300f3143e666232146dcaea" Oct 30 12:22:33.852767 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.852713 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c53e35dda16cea704270a009ae7fcfb50c40c5a6300f3143e666232146dcaea"} err="failed to get container status \"1c53e35dda16cea704270a009ae7fcfb50c40c5a6300f3143e666232146dcaea\": rpc error: code = NotFound desc = could not find container \"1c53e35dda16cea704270a009ae7fcfb50c40c5a6300f3143e666232146dcaea\": container with ID starting with 1c53e35dda16cea704270a009ae7fcfb50c40c5a6300f3143e666232146dcaea not found: ID does not exist" Oct 30 12:22:33.870688 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.870661 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:33.904353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.899402 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb"] Oct 30 12:22:33.904353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.903393 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7566f5-6d88789898-7t2nb"] Oct 30 12:22:33.990452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:33.990361 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:34.432727 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:34.432687 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:35.020785 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:35.020755 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50dada57-ddc4-4a37-bdd5-638e44f15a63" path="/var/lib/kubelet/pods/50dada57-ddc4-4a37-bdd5-638e44f15a63/volumes" Oct 30 12:22:35.672768 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:35.672722 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:35.866476 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:22:35.866435 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:22:37.937225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:37.937189 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:38.756064 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:38.756026 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:39.517524 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:39.517488 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:39.933038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:39.933003 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x"] Oct 30 12:22:39.934137 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:39.934109 4413 cpu_manager.go:414] "containerMap: removing container" podUID="50dada57-ddc4-4a37-bdd5-638e44f15a63" containerName="maas-api" Oct 30 12:22:39.934137 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:39.934133 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50dada57-ddc4-4a37-bdd5-638e44f15a63" containerName="maas-api" Oct 30 12:22:39.934416 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:39.934318 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="50dada57-ddc4-4a37-bdd5-638e44f15a63" containerName="maas-api" Oct 30 12:22:39.940085 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:39.940063 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:39.969310 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:39.969284 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8f2898-openshift-default-dockercfg-jqtp6\"" Oct 30 12:22:39.974145 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:39.974123 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x"] Oct 30 12:22:39.990449 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:39.990421 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:40.024961 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.024935 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-data\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.025094 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.024983 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-credential-socket\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.025165 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.025130 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-podinfo\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.025221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.025172 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-envoy\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.025305 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.025239 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlxlq\" (UniqueName: \"kubernetes.io/projected/8eefa82e-f061-4c58-8fad-46c497d22c93-kube-api-access-xlxlq\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.025397 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.025383 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eefa82e-f061-4c58-8fad-46c497d22c93-istiod-ca-cert\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.025447 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.025435 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-token\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.025501 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.025471 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-workload-socket\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.025501 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.025489 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-workload-certs\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.137280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.136543 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-workload-socket\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.137280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.136594 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-workload-certs\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.137280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.136773 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-data\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.137280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.136827 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-credential-socket\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.137280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.136949 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-podinfo\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.137280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.136975 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-envoy\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.137280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.137007 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlxlq\" (UniqueName: \"kubernetes.io/projected/8eefa82e-f061-4c58-8fad-46c497d22c93-kube-api-access-xlxlq\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.137280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.137128 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eefa82e-f061-4c58-8fad-46c497d22c93-istiod-ca-cert\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.137280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.137184 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-token\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.145272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.142451 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-workload-socket\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.145272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.142774 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-workload-certs\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.145272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.143262 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-data\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.145272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.143802 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-credential-socket\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.145272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.144453 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eefa82e-f061-4c58-8fad-46c497d22c93-istiod-ca-cert\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.145272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.144721 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:40.182502 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.182470 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-podinfo\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.197635 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.197557 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-envoy\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.214826 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.214793 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-token\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.221824 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.221610 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlxlq\" (UniqueName: \"kubernetes.io/projected/8eefa82e-f061-4c58-8fad-46c497d22c93-kube-api-access-xlxlq\") pod \"e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.254947 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.254913 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:40.265348 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.265319 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:40.595660 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.595631 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x"] Oct 30 12:22:40.596278 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:22:40.596233 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eefa82e_f061_4c58_8fad_46c497d22c93.slice/crio-9b34d77720f15fede23ee27086f29512161cc7a7c57296be5ff039421927cfb2 WatchSource:0}: Error finding container 9b34d77720f15fede23ee27086f29512161cc7a7c57296be5ff039421927cfb2: Status 404 returned error can't find the container with id 9b34d77720f15fede23ee27086f29512161cc7a7c57296be5ff039421927cfb2 Oct 30 12:22:40.599138 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.599103 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:22:40.599227 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.599176 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:22:40.599227 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.599205 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:22:40.901649 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.901611 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" event={"ID":"8eefa82e-f061-4c58-8fad-46c497d22c93","Type":"ContainerStarted","Data":"0a072c9ced35c544026c69ff8b64517f747cdfc8dd7728ee62797891110a61d4"} Oct 30 12:22:40.901845 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.901679 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" event={"ID":"8eefa82e-f061-4c58-8fad-46c497d22c93","Type":"ContainerStarted","Data":"9b34d77720f15fede23ee27086f29512161cc7a7c57296be5ff039421927cfb2"} Oct 30 12:22:40.948856 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:40.948804 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" podStartSLOduration=1.948788241 podStartE2EDuration="1.948788241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:40.94527604 +0000 UTC m=+1714.055118947" watchObservedRunningTime="2026-10-30 12:22:40.948788241 +0000 UTC m=+1714.058631122" Oct 30 12:22:41.121465 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:41.121424 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:41.243952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:41.243851 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:41.255597 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:41.255568 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:41.261583 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:41.261557 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:41.462267 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:41.462222 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:41.868147 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:41.868106 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:41.908542 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:41.908512 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:41.909609 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:41.909592 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:22:42.177586 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:42.176310 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:42.552468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:42.552380 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:43.627584 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:43.627547 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:43.742536 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:43.742506 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:44.555447 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:44.555415 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:45.519590 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:45.519552 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:45.634683 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:45.634650 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:46.378002 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.377969 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g"] Oct 30 12:22:46.378331 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.378276 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" podUID="4350a054-e8ff-435a-b68f-3b2a89bda23c" containerName="maas-api" containerID="cri-o://ecf289e679d42203fcab71dd662a3cb243bfd7a35b17ed15d9eb3015566fa853" gracePeriod=30 Oct 30 12:22:46.442719 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.442689 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:46.789630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.789600 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:22:46.881613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.879951 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4350a054-e8ff-435a-b68f-3b2a89bda23c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4350a054-e8ff-435a-b68f-3b2a89bda23c-openshift-service-ca\") pod \"4350a054-e8ff-435a-b68f-3b2a89bda23c\" (UID: \"4350a054-e8ff-435a-b68f-3b2a89bda23c\") " Oct 30 12:22:46.881613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.880031 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4350a054-e8ff-435a-b68f-3b2a89bda23c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4350a054-e8ff-435a-b68f-3b2a89bda23c-maas-api-tls\") pod \"4350a054-e8ff-435a-b68f-3b2a89bda23c\" (UID: \"4350a054-e8ff-435a-b68f-3b2a89bda23c\") " Oct 30 12:22:46.881613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.880061 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4350a054-e8ff-435a-b68f-3b2a89bda23c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4350a054-e8ff-435a-b68f-3b2a89bda23c-metrics-certs\") pod \"4350a054-e8ff-435a-b68f-3b2a89bda23c\" (UID: \"4350a054-e8ff-435a-b68f-3b2a89bda23c\") " Oct 30 12:22:46.881613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.880274 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4350a054-e8ff-435a-b68f-3b2a89bda23c-kube-api-access-rqq9v\" (UniqueName: \"kubernetes.io/projected/4350a054-e8ff-435a-b68f-3b2a89bda23c-kube-api-access-rqq9v\") pod \"4350a054-e8ff-435a-b68f-3b2a89bda23c\" (UID: \"4350a054-e8ff-435a-b68f-3b2a89bda23c\") " Oct 30 12:22:46.885882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.885846 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4350a054-e8ff-435a-b68f-3b2a89bda23c-openshift-service-ca" pod "4350a054-e8ff-435a-b68f-3b2a89bda23c" (UID: "4350a054-e8ff-435a-b68f-3b2a89bda23c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:22:46.895609 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.895548 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4350a054-e8ff-435a-b68f-3b2a89bda23c-metrics-certs" pod "4350a054-e8ff-435a-b68f-3b2a89bda23c" (UID: "4350a054-e8ff-435a-b68f-3b2a89bda23c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:22:46.896243 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.896201 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4350a054-e8ff-435a-b68f-3b2a89bda23c-maas-api-tls" pod "4350a054-e8ff-435a-b68f-3b2a89bda23c" (UID: "4350a054-e8ff-435a-b68f-3b2a89bda23c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:22:46.896931 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.896895 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4350a054-e8ff-435a-b68f-3b2a89bda23c-kube-api-access-rqq9v" pod "4350a054-e8ff-435a-b68f-3b2a89bda23c" (UID: "4350a054-e8ff-435a-b68f-3b2a89bda23c"). InnerVolumeSpecName "kube-api-access-rqq9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:22:46.980869 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.980836 4413 generic.go:356] "Generic (PLEG): container finished" podID="4350a054-e8ff-435a-b68f-3b2a89bda23c" containerID="ecf289e679d42203fcab71dd662a3cb243bfd7a35b17ed15d9eb3015566fa853" exitCode=0 Oct 30 12:22:46.983030 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.980894 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" event={"ID":"4350a054-e8ff-435a-b68f-3b2a89bda23c","Type":"ContainerDied","Data":"ecf289e679d42203fcab71dd662a3cb243bfd7a35b17ed15d9eb3015566fa853"} Oct 30 12:22:46.983030 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.982345 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" event={"ID":"4350a054-e8ff-435a-b68f-3b2a89bda23c","Type":"ContainerDied","Data":"4b1c9042de21acea33a9f5766bd4804cb28e15d4b9858aab0b6556ada4d31069"} Oct 30 12:22:46.983030 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.982423 4413 scope.go:122] "RemoveContainer" containerID="ecf289e679d42203fcab71dd662a3cb243bfd7a35b17ed15d9eb3015566fa853" Oct 30 12:22:46.983030 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.980925 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g" Oct 30 12:22:46.985847 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.985826 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4350a054-e8ff-435a-b68f-3b2a89bda23c-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:22:46.985971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.985851 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4350a054-e8ff-435a-b68f-3b2a89bda23c-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:22:46.985971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.985867 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqq9v\" (UniqueName: \"kubernetes.io/projected/4350a054-e8ff-435a-b68f-3b2a89bda23c-kube-api-access-rqq9v\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:22:46.985971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:46.985882 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4350a054-e8ff-435a-b68f-3b2a89bda23c-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:22:47.001404 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:47.001366 4413 scope.go:122] "RemoveContainer" containerID="ecf289e679d42203fcab71dd662a3cb243bfd7a35b17ed15d9eb3015566fa853" Oct 30 12:22:47.003059 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:22:47.001871 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf289e679d42203fcab71dd662a3cb243bfd7a35b17ed15d9eb3015566fa853\": container with ID starting with ecf289e679d42203fcab71dd662a3cb243bfd7a35b17ed15d9eb3015566fa853 not found: ID does not exist" containerID="ecf289e679d42203fcab71dd662a3cb243bfd7a35b17ed15d9eb3015566fa853" Oct 30 12:22:47.003059 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:47.001907 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf289e679d42203fcab71dd662a3cb243bfd7a35b17ed15d9eb3015566fa853"} err="failed to get container status \"ecf289e679d42203fcab71dd662a3cb243bfd7a35b17ed15d9eb3015566fa853\": rpc error: code = NotFound desc = could not find container \"ecf289e679d42203fcab71dd662a3cb243bfd7a35b17ed15d9eb3015566fa853\": container with ID starting with ecf289e679d42203fcab71dd662a3cb243bfd7a35b17ed15d9eb3015566fa853 not found: ID does not exist" Oct 30 12:22:47.040038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:47.040002 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g"] Oct 30 12:22:47.045365 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:47.045338 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82299a-6659bf8dd9-b426g"] Oct 30 12:22:47.301791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:47.301710 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:47.447574 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:47.447544 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:47.562704 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:47.562584 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:47.996437 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:47.996404 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:49.020866 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:49.020834 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4350a054-e8ff-435a-b68f-3b2a89bda23c" path="/var/lib/kubelet/pods/4350a054-e8ff-435a-b68f-3b2a89bda23c/volumes" Oct 30 12:22:49.370173 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:49.370095 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:51.180453 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:51.180416 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:52.903708 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:52.903677 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:54.285714 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:54.285676 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:54.498742 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:54.498704 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:54.685516 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:54.685483 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9"] Oct 30 12:22:54.685999 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:54.685966 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" podUID="42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" containerName="istio-proxy" containerID="cri-o://cbb155d60daf1bdc1959aa3db9a7cc0d158dd3eeb3030a8acf845740d0d4c5f3" gracePeriod=30 Oct 30 12:22:54.737159 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:54.737115 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:54.892277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:54.891921 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:56.854356 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:56.854319 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:57.017065 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:57.016980 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:58.771765 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:58.771728 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:22:59.971737 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.971683 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:22:59.991374 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.991349 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-data\") pod \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " Oct 30 12:22:59.991491 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.991420 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-podinfo\") pod \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " Oct 30 12:22:59.991543 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.991490 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istiod-ca-cert\") pod \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " Oct 30 12:22:59.991585 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.991560 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-credential-socket\") pod \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " Oct 30 12:22:59.991631 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.991590 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-kube-api-access-s77pc\" (UniqueName: \"kubernetes.io/projected/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-kube-api-access-s77pc\") pod \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " Oct 30 12:22:59.991631 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.991600 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-data" pod "42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" (UID: "42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:22:59.991709 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.991635 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-envoy\") pod \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " Oct 30 12:22:59.991854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.991813 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-credential-socket" pod "42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" (UID: "42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:22:59.991854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.991837 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istiod-ca-cert" pod "42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" (UID: "42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:22:59.991854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.991840 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-workload-socket\") pod \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " Oct 30 12:22:59.992057 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.991926 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-workload-certs\") pod \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " Oct 30 12:22:59.992057 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.991952 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-token\" (UniqueName: \"kubernetes.io/projected/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-token\") pod \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\" (UID: \"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d\") " Oct 30 12:22:59.992161 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.992061 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-workload-socket" pod "42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" (UID: "42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:22:59.992550 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.992528 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-workload-certs" pod "42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" (UID: "42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:22:59.992817 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.992798 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:22:59.992926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.992821 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:22:59.992926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.992835 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:22:59.992926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.992848 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:22:59.992926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.992861 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:22:59.994365 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.994337 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-kube-api-access-s77pc" pod "42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" (UID: "42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d"). InnerVolumeSpecName "kube-api-access-s77pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:22:59.994458 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.994385 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-envoy" pod "42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" (UID: "42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:22:59.994458 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.994389 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-podinfo" pod "42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" (UID: "42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:22:59.994606 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:22:59.994590 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-token" pod "42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" (UID: "42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:23:00.074911 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.074878 4413 generic.go:356] "Generic (PLEG): container finished" podID="42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" containerID="cbb155d60daf1bdc1959aa3db9a7cc0d158dd3eeb3030a8acf845740d0d4c5f3" exitCode=0 Oct 30 12:23:00.075086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.074945 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" Oct 30 12:23:00.075086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.074967 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" event={"ID":"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d","Type":"ContainerDied","Data":"cbb155d60daf1bdc1959aa3db9a7cc0d158dd3eeb3030a8acf845740d0d4c5f3"} Oct 30 12:23:00.075086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.075035 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9" event={"ID":"42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d","Type":"ContainerDied","Data":"8678aab983f6272efca3057f6b075f3aeafd14482599b179928de324ece5b7bd"} Oct 30 12:23:00.075086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.075067 4413 scope.go:122] "RemoveContainer" containerID="cbb155d60daf1bdc1959aa3db9a7cc0d158dd3eeb3030a8acf845740d0d4c5f3" Oct 30 12:23:00.090699 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.090677 4413 scope.go:122] "RemoveContainer" containerID="97b96593d501ed9d9131cdf7055ed4baa97c4a68c19b584b1256cb90725bd513" Oct 30 12:23:00.094433 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.094409 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:00.094560 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.094436 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:00.094560 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.094450 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s77pc\" (UniqueName: \"kubernetes.io/projected/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-kube-api-access-s77pc\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:00.094560 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.094464 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:00.100392 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.100366 4413 scope.go:122] "RemoveContainer" containerID="cbb155d60daf1bdc1959aa3db9a7cc0d158dd3eeb3030a8acf845740d0d4c5f3" Oct 30 12:23:00.100697 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:23:00.100668 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb155d60daf1bdc1959aa3db9a7cc0d158dd3eeb3030a8acf845740d0d4c5f3\": container with ID starting with cbb155d60daf1bdc1959aa3db9a7cc0d158dd3eeb3030a8acf845740d0d4c5f3 not found: ID does not exist" containerID="cbb155d60daf1bdc1959aa3db9a7cc0d158dd3eeb3030a8acf845740d0d4c5f3" Oct 30 12:23:00.100759 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.100701 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb155d60daf1bdc1959aa3db9a7cc0d158dd3eeb3030a8acf845740d0d4c5f3"} err="failed to get container status \"cbb155d60daf1bdc1959aa3db9a7cc0d158dd3eeb3030a8acf845740d0d4c5f3\": rpc error: code = NotFound desc = could not find container \"cbb155d60daf1bdc1959aa3db9a7cc0d158dd3eeb3030a8acf845740d0d4c5f3\": container with ID starting with cbb155d60daf1bdc1959aa3db9a7cc0d158dd3eeb3030a8acf845740d0d4c5f3 not found: ID does not exist" Oct 30 12:23:00.100759 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.100719 4413 scope.go:122] "RemoveContainer" containerID="97b96593d501ed9d9131cdf7055ed4baa97c4a68c19b584b1256cb90725bd513" Oct 30 12:23:00.101012 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:23:00.100989 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b96593d501ed9d9131cdf7055ed4baa97c4a68c19b584b1256cb90725bd513\": container with ID starting with 97b96593d501ed9d9131cdf7055ed4baa97c4a68c19b584b1256cb90725bd513 not found: ID does not exist" containerID="97b96593d501ed9d9131cdf7055ed4baa97c4a68c19b584b1256cb90725bd513" Oct 30 12:23:00.101097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.101017 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b96593d501ed9d9131cdf7055ed4baa97c4a68c19b584b1256cb90725bd513"} err="failed to get container status \"97b96593d501ed9d9131cdf7055ed4baa97c4a68c19b584b1256cb90725bd513\": rpc error: code = NotFound desc = could not find container \"97b96593d501ed9d9131cdf7055ed4baa97c4a68c19b584b1256cb90725bd513\": container with ID starting with 97b96593d501ed9d9131cdf7055ed4baa97c4a68c19b584b1256cb90725bd513 not found: ID does not exist" Oct 30 12:23:00.130238 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.130205 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9"] Oct 30 12:23:00.136406 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.136385 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-82299a-openshift-default-67884c4f67-nlnl9"] Oct 30 12:23:00.581823 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.581793 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j"] Oct 30 12:23:00.582467 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.582454 4413 cpu_manager.go:414] "containerMap: removing container" podUID="42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" containerName="istio-proxy" Oct 30 12:23:00.582516 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.582469 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" containerName="istio-proxy" Oct 30 12:23:00.582516 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.582481 4413 cpu_manager.go:414] "containerMap: removing container" podUID="42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" containerName="istio-proxy" Oct 30 12:23:00.582516 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.582486 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" containerName="istio-proxy" Oct 30 12:23:00.582516 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.582500 4413 cpu_manager.go:414] "containerMap: removing container" podUID="4350a054-e8ff-435a-b68f-3b2a89bda23c" containerName="maas-api" Oct 30 12:23:00.582516 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.582505 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4350a054-e8ff-435a-b68f-3b2a89bda23c" containerName="maas-api" Oct 30 12:23:00.582672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.582595 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="4350a054-e8ff-435a-b68f-3b2a89bda23c" containerName="maas-api" Oct 30 12:23:00.582672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.582611 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" containerName="istio-proxy" Oct 30 12:23:00.582672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.582619 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" containerName="istio-proxy" Oct 30 12:23:00.587579 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.587564 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:23:00.600864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.600840 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661c97b4-755e-44af-b5bb-bf67720638bb-metrics-certs\") pod \"maas-api-e2e-api-b-8f2898-746b477794-njl5j\" (UID: \"661c97b4-755e-44af-b5bb-bf67720638bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:23:00.601078 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.601052 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661c97b4-755e-44af-b5bb-bf67720638bb-openshift-service-ca\") pod \"maas-api-e2e-api-b-8f2898-746b477794-njl5j\" (UID: \"661c97b4-755e-44af-b5bb-bf67720638bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:23:00.601296 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.601268 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661c97b4-755e-44af-b5bb-bf67720638bb-maas-api-tls\") pod \"maas-api-e2e-api-b-8f2898-746b477794-njl5j\" (UID: \"661c97b4-755e-44af-b5bb-bf67720638bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:23:00.601400 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.601333 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p8cj\" (UniqueName: \"kubernetes.io/projected/661c97b4-755e-44af-b5bb-bf67720638bb-kube-api-access-7p8cj\") pod \"maas-api-e2e-api-b-8f2898-746b477794-njl5j\" (UID: \"661c97b4-755e-44af-b5bb-bf67720638bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:23:00.601841 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.601823 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8f2898-serving-cert\"" Oct 30 12:23:00.607548 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.607525 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j"] Oct 30 12:23:00.674084 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.674052 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:00.703222 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.703195 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661c97b4-755e-44af-b5bb-bf67720638bb-openshift-service-ca\") pod \"maas-api-e2e-api-b-8f2898-746b477794-njl5j\" (UID: \"661c97b4-755e-44af-b5bb-bf67720638bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:23:00.703362 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.703230 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661c97b4-755e-44af-b5bb-bf67720638bb-maas-api-tls\") pod \"maas-api-e2e-api-b-8f2898-746b477794-njl5j\" (UID: \"661c97b4-755e-44af-b5bb-bf67720638bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:23:00.703362 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.703292 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p8cj\" (UniqueName: \"kubernetes.io/projected/661c97b4-755e-44af-b5bb-bf67720638bb-kube-api-access-7p8cj\") pod \"maas-api-e2e-api-b-8f2898-746b477794-njl5j\" (UID: \"661c97b4-755e-44af-b5bb-bf67720638bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:23:00.703448 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:23:00.703433 4413 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-serving-cert: secret "maas-api-e2e-api-b-8f2898-serving-cert" not found Oct 30 12:23:00.703511 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:23:00.703498 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661c97b4-755e-44af-b5bb-bf67720638bb-maas-api-tls podName:661c97b4-755e-44af-b5bb-bf67720638bb nodeName:}" failed. No retries permitted until 2026-10-30 12:23:01.203479704 +0000 UTC m=+1734.313322564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/661c97b4-755e-44af-b5bb-bf67720638bb-maas-api-tls") pod "maas-api-e2e-api-b-8f2898-746b477794-njl5j" (UID: "661c97b4-755e-44af-b5bb-bf67720638bb") : secret "maas-api-e2e-api-b-8f2898-serving-cert" not found Oct 30 12:23:00.703564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.703442 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661c97b4-755e-44af-b5bb-bf67720638bb-metrics-certs\") pod \"maas-api-e2e-api-b-8f2898-746b477794-njl5j\" (UID: \"661c97b4-755e-44af-b5bb-bf67720638bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:23:00.703892 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.703873 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661c97b4-755e-44af-b5bb-bf67720638bb-openshift-service-ca\") pod \"maas-api-e2e-api-b-8f2898-746b477794-njl5j\" (UID: \"661c97b4-755e-44af-b5bb-bf67720638bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:23:00.706138 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.706114 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661c97b4-755e-44af-b5bb-bf67720638bb-metrics-certs\") pod \"maas-api-e2e-api-b-8f2898-746b477794-njl5j\" (UID: \"661c97b4-755e-44af-b5bb-bf67720638bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:23:00.727005 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.726972 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p8cj\" (UniqueName: \"kubernetes.io/projected/661c97b4-755e-44af-b5bb-bf67720638bb-kube-api-access-7p8cj\") pod \"maas-api-e2e-api-b-8f2898-746b477794-njl5j\" (UID: \"661c97b4-755e-44af-b5bb-bf67720638bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:23:00.787942 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.787902 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:00.920841 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:00.920806 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:01.020703 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:01.020669 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d" path="/var/lib/kubelet/pods/42d4ab61-5cb6-4c3f-9692-ece09ff0bd3d/volumes" Oct 30 12:23:01.210794 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:01.210717 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661c97b4-755e-44af-b5bb-bf67720638bb-maas-api-tls\") pod \"maas-api-e2e-api-b-8f2898-746b477794-njl5j\" (UID: \"661c97b4-755e-44af-b5bb-bf67720638bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:23:01.213436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:01.213413 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661c97b4-755e-44af-b5bb-bf67720638bb-maas-api-tls\") pod \"maas-api-e2e-api-b-8f2898-746b477794-njl5j\" (UID: \"661c97b4-755e-44af-b5bb-bf67720638bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:23:01.471477 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:01.471388 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:01.501653 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:01.501624 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:23:01.599472 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:01.599385 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:01.845550 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:01.823151 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j"] Oct 30 12:23:01.851060 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:23:01.851006 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661c97b4_755e_44af_b5bb_bf67720638bb.slice/crio-2ed82d7881d96377dc87fe50f750eeead0d882bd516a40980c12df362dcd2f4a WatchSource:0}: Error finding container 2ed82d7881d96377dc87fe50f750eeead0d882bd516a40980c12df362dcd2f4a: Status 404 returned error can't find the container with id 2ed82d7881d96377dc87fe50f750eeead0d882bd516a40980c12df362dcd2f4a Oct 30 12:23:02.093504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:02.093407 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" event={"ID":"661c97b4-755e-44af-b5bb-bf67720638bb","Type":"ContainerStarted","Data":"2ed82d7881d96377dc87fe50f750eeead0d882bd516a40980c12df362dcd2f4a"} Oct 30 12:23:02.855796 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:02.855763 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:03.102013 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:03.101979 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" event={"ID":"661c97b4-755e-44af-b5bb-bf67720638bb","Type":"ContainerStarted","Data":"a39c29425bed5b3ded1fd6da679612b782ac444a855b76fbbe528b5f7296c1df"} Oct 30 12:23:03.102497 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:03.102090 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:23:03.148153 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:03.148097 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" podStartSLOduration=2.8242872820000002 podStartE2EDuration="3.148082397s" totalImagesPullingTime="323.795115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:23:03.144347344 +0000 UTC m=+1736.254190225" watchObservedRunningTime="2026-10-30 12:23:03.148082397 +0000 UTC m=+1736.257925302" Oct 30 12:23:03.471226 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:03.471131 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:04.656475 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:04.656442 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:05.777586 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:05.777551 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk"] Oct 30 12:23:05.816609 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:23:05.816575 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:23:05.946973 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:05.946940 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:06.056036 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:06.055954 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:06.468272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:06.465730 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:08.261331 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:08.261297 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:09.113807 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:09.113775 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:23:09.723667 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:09.723633 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:09.949523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:09.949494 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:10.777943 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:10.777892 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk" podUID="b401ad16-659d-472c-9934-26785d401eff" containerName="payload-processing" containerID="cri-o://525edd9edc00f361a0a0a9218040f5c8d30bc3db85f6f01abe5adf08af09405b" gracePeriod=25 Oct 30 12:23:11.065088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:11.065067 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk" Oct 30 12:23:11.110449 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:11.110424 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b401ad16-659d-472c-9934-26785d401eff-kube-api-access-9mwpr\" (UniqueName: \"kubernetes.io/projected/b401ad16-659d-472c-9934-26785d401eff-kube-api-access-9mwpr\") pod \"b401ad16-659d-472c-9934-26785d401eff\" (UID: \"b401ad16-659d-472c-9934-26785d401eff\") " Oct 30 12:23:11.110656 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:11.110630 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b401ad16-659d-472c-9934-26785d401eff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b401ad16-659d-472c-9934-26785d401eff-plugins-config-volume\") pod \"b401ad16-659d-472c-9934-26785d401eff\" (UID: \"b401ad16-659d-472c-9934-26785d401eff\") " Oct 30 12:23:11.110997 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:11.110974 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b401ad16-659d-472c-9934-26785d401eff-plugins-config-volume" pod "b401ad16-659d-472c-9934-26785d401eff" (UID: "b401ad16-659d-472c-9934-26785d401eff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:23:11.111401 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:11.111385 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b401ad16-659d-472c-9934-26785d401eff-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:11.112965 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:11.112940 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b401ad16-659d-472c-9934-26785d401eff-kube-api-access-9mwpr" pod "b401ad16-659d-472c-9934-26785d401eff" (UID: "b401ad16-659d-472c-9934-26785d401eff"). InnerVolumeSpecName "kube-api-access-9mwpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:23:11.160178 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:11.160146 4413 generic.go:356] "Generic (PLEG): container finished" podID="b401ad16-659d-472c-9934-26785d401eff" containerID="525edd9edc00f361a0a0a9218040f5c8d30bc3db85f6f01abe5adf08af09405b" exitCode=0 Oct 30 12:23:11.160545 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:11.160243 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk" event={"ID":"b401ad16-659d-472c-9934-26785d401eff","Type":"ContainerDied","Data":"525edd9edc00f361a0a0a9218040f5c8d30bc3db85f6f01abe5adf08af09405b"} Oct 30 12:23:11.160663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:11.160637 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk" event={"ID":"b401ad16-659d-472c-9934-26785d401eff","Type":"ContainerDied","Data":"1109b6dfd59f8907a9ee1a6ff13cbe061377af5cbf081df8972e92dcc8524c89"} Oct 30 12:23:11.160734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:11.160355 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk" Oct 30 12:23:11.160734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:11.160687 4413 scope.go:122] "RemoveContainer" containerID="525edd9edc00f361a0a0a9218040f5c8d30bc3db85f6f01abe5adf08af09405b" Oct 30 12:23:11.172590 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:11.172568 4413 scope.go:122] "RemoveContainer" containerID="525edd9edc00f361a0a0a9218040f5c8d30bc3db85f6f01abe5adf08af09405b" Oct 30 12:23:11.172981 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:23:11.172910 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"525edd9edc00f361a0a0a9218040f5c8d30bc3db85f6f01abe5adf08af09405b\": container with ID starting with 525edd9edc00f361a0a0a9218040f5c8d30bc3db85f6f01abe5adf08af09405b not found: ID does not exist" containerID="525edd9edc00f361a0a0a9218040f5c8d30bc3db85f6f01abe5adf08af09405b" Oct 30 12:23:11.172981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:11.172948 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525edd9edc00f361a0a0a9218040f5c8d30bc3db85f6f01abe5adf08af09405b"} err="failed to get container status \"525edd9edc00f361a0a0a9218040f5c8d30bc3db85f6f01abe5adf08af09405b\": rpc error: code = NotFound desc = could not find container \"525edd9edc00f361a0a0a9218040f5c8d30bc3db85f6f01abe5adf08af09405b\": container with ID starting with 525edd9edc00f361a0a0a9218040f5c8d30bc3db85f6f01abe5adf08af09405b not found: ID does not exist" Oct 30 12:23:11.212917 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:11.212884 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk"] Oct 30 12:23:11.213053 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:11.212963 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mwpr\" (UniqueName: \"kubernetes.io/projected/b401ad16-659d-472c-9934-26785d401eff-kube-api-access-9mwpr\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:11.217184 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:11.217165 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ebc6bb-768fc47bb4-x2rdk"] Oct 30 12:23:11.588874 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:11.588840 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:13.020505 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:13.020474 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b401ad16-659d-472c-9934-26785d401eff" path="/var/lib/kubelet/pods/b401ad16-659d-472c-9934-26785d401eff/volumes" Oct 30 12:23:13.274548 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:13.274478 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:14.667183 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:14.667149 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg"] Oct 30 12:23:14.667673 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:14.667418 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" podUID="cb6b5578-4eac-409a-a8fb-f174ad8aeeb2" containerName="maas-api" containerID="cri-o://48d04cef3fa97257a7e5ab674c40c93b2622b3462ebc7b61bde74063eb8653d7" gracePeriod=30 Oct 30 12:23:14.777517 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:14.777484 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:15.013120 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.013097 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:23:15.067394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.067361 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-metrics-certs\") pod \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\" (UID: \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\") " Oct 30 12:23:15.067576 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.067532 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-openshift-service-ca\") pod \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\" (UID: \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\") " Oct 30 12:23:15.067649 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.067596 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-maas-api-tls\") pod \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\" (UID: \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\") " Oct 30 12:23:15.067799 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.067776 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-kube-api-access-l8cmv\" (UniqueName: \"kubernetes.io/projected/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-kube-api-access-l8cmv\") pod \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\" (UID: \"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2\") " Oct 30 12:23:15.069675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.069642 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-openshift-service-ca" pod "cb6b5578-4eac-409a-a8fb-f174ad8aeeb2" (UID: "cb6b5578-4eac-409a-a8fb-f174ad8aeeb2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:23:15.070281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.070243 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-metrics-certs" pod "cb6b5578-4eac-409a-a8fb-f174ad8aeeb2" (UID: "cb6b5578-4eac-409a-a8fb-f174ad8aeeb2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:23:15.072589 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.072566 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-maas-api-tls" pod "cb6b5578-4eac-409a-a8fb-f174ad8aeeb2" (UID: "cb6b5578-4eac-409a-a8fb-f174ad8aeeb2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:23:15.088429 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.088400 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-kube-api-access-l8cmv" pod "cb6b5578-4eac-409a-a8fb-f174ad8aeeb2" (UID: "cb6b5578-4eac-409a-a8fb-f174ad8aeeb2"). InnerVolumeSpecName "kube-api-access-l8cmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:23:15.174313 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.173632 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:15.174313 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.173684 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:15.174313 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.173700 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8cmv\" (UniqueName: \"kubernetes.io/projected/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-kube-api-access-l8cmv\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:15.174313 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.173714 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:15.190829 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.190792 4413 generic.go:356] "Generic (PLEG): container finished" podID="cb6b5578-4eac-409a-a8fb-f174ad8aeeb2" containerID="48d04cef3fa97257a7e5ab674c40c93b2622b3462ebc7b61bde74063eb8653d7" exitCode=0 Oct 30 12:23:15.191293 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.191261 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" event={"ID":"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2","Type":"ContainerDied","Data":"48d04cef3fa97257a7e5ab674c40c93b2622b3462ebc7b61bde74063eb8653d7"} Oct 30 12:23:15.192784 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.192758 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" event={"ID":"cb6b5578-4eac-409a-a8fb-f174ad8aeeb2","Type":"ContainerDied","Data":"5afa094eabb2156a655a4643cdf3a87f14ad6e3644a524d254cc1aa75fe4b71d"} Oct 30 12:23:15.194228 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.194203 4413 scope.go:122] "RemoveContainer" containerID="48d04cef3fa97257a7e5ab674c40c93b2622b3462ebc7b61bde74063eb8653d7" Oct 30 12:23:15.194424 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.194409 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg" Oct 30 12:23:15.212176 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.212153 4413 scope.go:122] "RemoveContainer" containerID="48d04cef3fa97257a7e5ab674c40c93b2622b3462ebc7b61bde74063eb8653d7" Oct 30 12:23:15.212641 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:23:15.212611 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d04cef3fa97257a7e5ab674c40c93b2622b3462ebc7b61bde74063eb8653d7\": container with ID starting with 48d04cef3fa97257a7e5ab674c40c93b2622b3462ebc7b61bde74063eb8653d7 not found: ID does not exist" containerID="48d04cef3fa97257a7e5ab674c40c93b2622b3462ebc7b61bde74063eb8653d7" Oct 30 12:23:15.212788 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.212654 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d04cef3fa97257a7e5ab674c40c93b2622b3462ebc7b61bde74063eb8653d7"} err="failed to get container status \"48d04cef3fa97257a7e5ab674c40c93b2622b3462ebc7b61bde74063eb8653d7\": rpc error: code = NotFound desc = could not find container \"48d04cef3fa97257a7e5ab674c40c93b2622b3462ebc7b61bde74063eb8653d7\": container with ID starting with 48d04cef3fa97257a7e5ab674c40c93b2622b3462ebc7b61bde74063eb8653d7 not found: ID does not exist" Oct 30 12:23:15.252138 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.252047 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg"] Oct 30 12:23:15.255482 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.255450 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-060f41f2-744745b898-zg6cg"] Oct 30 12:23:15.277952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.277879 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:15.403594 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.403562 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:15.799850 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:15.799811 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:16.010956 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:16.010926 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:17.023092 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:17.023061 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb6b5578-4eac-409a-a8fb-f174ad8aeeb2" path="/var/lib/kubelet/pods/cb6b5578-4eac-409a-a8fb-f174ad8aeeb2/volumes" Oct 30 12:23:17.719380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:17.719336 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:19.034224 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:19.034155 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:19.544353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:19.544326 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:20.970751 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:20.970720 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5"] Oct 30 12:23:21.041855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:21.041822 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:35.814546 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:23:35.814512 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:23:35.993718 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:35.993680 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" podUID="86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183" containerName="main" containerID="cri-o://36083e1551e2caafbe1076dfe8566495a6514df884d67614ef57274702001629" gracePeriod=45 Oct 30 12:23:36.288298 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.288264 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:23:36.364816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.364722 4413 generic.go:356] "Generic (PLEG): container finished" podID="86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183" containerID="36083e1551e2caafbe1076dfe8566495a6514df884d67614ef57274702001629" exitCode=0 Oct 30 12:23:36.364816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.364789 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" event={"ID":"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183","Type":"ContainerDied","Data":"36083e1551e2caafbe1076dfe8566495a6514df884d67614ef57274702001629"} Oct 30 12:23:36.365040 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.364830 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" event={"ID":"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183","Type":"ContainerDied","Data":"6a1e3e4a7f00e2d7595edeafd5f9f54a98de4d5b35350e2aad87c0568a914fd2"} Oct 30 12:23:36.365040 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.364855 4413 scope.go:122] "RemoveContainer" containerID="36083e1551e2caafbe1076dfe8566495a6514df884d67614ef57274702001629" Oct 30 12:23:36.365040 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.364790 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5" Oct 30 12:23:36.378903 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.378878 4413 scope.go:122] "RemoveContainer" containerID="36083e1551e2caafbe1076dfe8566495a6514df884d67614ef57274702001629" Oct 30 12:23:36.384652 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:23:36.384618 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36083e1551e2caafbe1076dfe8566495a6514df884d67614ef57274702001629\": container with ID starting with 36083e1551e2caafbe1076dfe8566495a6514df884d67614ef57274702001629 not found: ID does not exist" containerID="36083e1551e2caafbe1076dfe8566495a6514df884d67614ef57274702001629" Oct 30 12:23:36.384771 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.384664 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36083e1551e2caafbe1076dfe8566495a6514df884d67614ef57274702001629"} err="failed to get container status \"36083e1551e2caafbe1076dfe8566495a6514df884d67614ef57274702001629\": rpc error: code = NotFound desc = could not find container \"36083e1551e2caafbe1076dfe8566495a6514df884d67614ef57274702001629\": container with ID starting with 36083e1551e2caafbe1076dfe8566495a6514df884d67614ef57274702001629 not found: ID does not exist" Oct 30 12:23:36.416846 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.416814 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-tls-certs\" (UniqueName: \"kubernetes.io/secret/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-tls-certs\") pod \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " Oct 30 12:23:36.416981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.416914 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-kube-api-access-lk4fj\" (UniqueName: \"kubernetes.io/projected/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-kube-api-access-lk4fj\") pod \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " Oct 30 12:23:36.417033 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.416982 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-dshm\") pod \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " Oct 30 12:23:36.417129 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.417115 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-tmp-dir\") pod \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " Oct 30 12:23:36.417213 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.417198 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-model-cache\") pod \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " Oct 30 12:23:36.417278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.417268 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-home\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-home\") pod \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\" (UID: \"86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183\") " Oct 30 12:23:36.417464 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.417440 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-tmp-dir" pod "86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183" (UID: "86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:23:36.417562 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.417492 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-model-cache" pod "86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183" (UID: "86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:23:36.417774 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.417748 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-home" pod "86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183" (UID: "86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:23:36.418029 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.418011 4413 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:36.418082 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.418038 4413 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:36.418082 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.418056 4413 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:36.419505 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.419484 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-kube-api-access-lk4fj" pod "86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183" (UID: "86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183"). InnerVolumeSpecName "kube-api-access-lk4fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:23:36.419582 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.419508 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-tls-certs" pod "86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183" (UID: "86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:23:36.419666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.419652 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-dshm" pod "86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183" (UID: "86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:23:36.519342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.519308 4413 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:36.519342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.519339 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk4fj\" (UniqueName: \"kubernetes.io/projected/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-kube-api-access-lk4fj\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:36.519342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.519348 4413 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:36.711067 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.711039 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5"] Oct 30 12:23:36.715455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:36.715431 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-060f41f2/e2e-lifecycle-model-060f41f2-kserve-5d8c855ccc-n4pj5"] Oct 30 12:23:37.020749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:37.020710 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183" path="/var/lib/kubelet/pods/86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183/volumes" Oct 30 12:23:37.374577 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:37.374547 4413 generic.go:356] "Generic (PLEG): container finished" podID="8eefa82e-f061-4c58-8fad-46c497d22c93" containerID="0a072c9ced35c544026c69ff8b64517f747cdfc8dd7728ee62797891110a61d4" exitCode=0 Oct 30 12:23:37.374726 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:37.374584 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" event={"ID":"8eefa82e-f061-4c58-8fad-46c497d22c93","Type":"ContainerDied","Data":"0a072c9ced35c544026c69ff8b64517f747cdfc8dd7728ee62797891110a61d4"} Oct 30 12:23:37.375005 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:37.374988 4413 scope.go:122] "RemoveContainer" containerID="0a072c9ced35c544026c69ff8b64517f747cdfc8dd7728ee62797891110a61d4" Oct 30 12:23:37.376487 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:37.376461 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:23:37.376547 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:37.376529 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:23:37.376581 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:37.376560 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:23:38.384512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:38.384479 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" event={"ID":"8eefa82e-f061-4c58-8fad-46c497d22c93","Type":"ContainerStarted","Data":"373d32224d2508b0fd25b1f1d8ced66b9b9288be9ccebc9e2c4fd2d40e99e54c"} Oct 30 12:23:39.255952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:39.255917 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:23:39.256872 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:39.256849 4413 patch_prober.go:29] interesting pod/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 12:23:39.256937 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:39.256903 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" podUID="8eefa82e-f061-4c58-8fad-46c497d22c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 12:23:40.256374 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:40.256340 4413 patch_prober.go:29] interesting pod/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 12:23:40.256783 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:40.256401 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" podUID="8eefa82e-f061-4c58-8fad-46c497d22c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 12:23:41.256390 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:41.256353 4413 patch_prober.go:29] interesting pod/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 12:23:41.256790 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:41.256414 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" podUID="8eefa82e-f061-4c58-8fad-46c497d22c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 12:23:41.911275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:41.909345 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:23:42.256501 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:42.256420 4413 patch_prober.go:29] interesting pod/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 12:23:42.256862 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:42.256498 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" podUID="8eefa82e-f061-4c58-8fad-46c497d22c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 12:23:43.256381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:43.256346 4413 patch_prober.go:29] interesting pod/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 12:23:43.256557 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:43.256408 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" podUID="8eefa82e-f061-4c58-8fad-46c497d22c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 12:23:43.690875 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:43.690836 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:44.260157 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:44.260124 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:23:44.261366 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:44.261345 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:23:50.977785 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:50.977692 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:51.622414 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:51.622383 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx"] Oct 30 12:23:51.636855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:51.636826 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw"] Oct 30 12:23:51.749181 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:51.749145 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:51.871555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:51.871525 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:52.956692 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:52.956658 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:54.757939 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:54.757904 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:56.204669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:56.204638 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:56.623404 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:56.623367 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx" podUID="68d6d475-531b-435f-b15b-65e7ed8437b4" containerName="payload-processing" containerID="cri-o://4cc4b274e933585e28e753e0118d48114da528bd78c991d03037f76d90abe77f" gracePeriod=25 Oct 30 12:23:56.637856 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:56.637819 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw" podUID="064df158-8c5e-4cab-bef7-9b9e2a64f460" containerName="payload-pre-processing" containerID="cri-o://a3953db329e7e9f2493eaeec5557c3d8dba577a12c24914c5332ef6ece4fbe52" gracePeriod=25 Oct 30 12:23:56.898938 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:56.898805 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:56.969481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:56.969459 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx" Oct 30 12:23:56.988551 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:56.988531 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw" Oct 30 12:23:57.046096 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.046062 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d6d475-531b-435f-b15b-65e7ed8437b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d6d475-531b-435f-b15b-65e7ed8437b4-plugins-config-volume\") pod \"68d6d475-531b-435f-b15b-65e7ed8437b4\" (UID: \"68d6d475-531b-435f-b15b-65e7ed8437b4\") " Oct 30 12:23:57.046302 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.046185 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d6d475-531b-435f-b15b-65e7ed8437b4-kube-api-access-sk6k6\" (UniqueName: \"kubernetes.io/projected/68d6d475-531b-435f-b15b-65e7ed8437b4-kube-api-access-sk6k6\") pod \"68d6d475-531b-435f-b15b-65e7ed8437b4\" (UID: \"68d6d475-531b-435f-b15b-65e7ed8437b4\") " Oct 30 12:23:57.046302 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.046287 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/064df158-8c5e-4cab-bef7-9b9e2a64f460-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064df158-8c5e-4cab-bef7-9b9e2a64f460-plugins-config-volume\") pod \"064df158-8c5e-4cab-bef7-9b9e2a64f460\" (UID: \"064df158-8c5e-4cab-bef7-9b9e2a64f460\") " Oct 30 12:23:57.046488 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.046468 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/064df158-8c5e-4cab-bef7-9b9e2a64f460-kube-api-access-z9n89\" (UniqueName: \"kubernetes.io/projected/064df158-8c5e-4cab-bef7-9b9e2a64f460-kube-api-access-z9n89\") pod \"064df158-8c5e-4cab-bef7-9b9e2a64f460\" (UID: \"064df158-8c5e-4cab-bef7-9b9e2a64f460\") " Oct 30 12:23:57.046877 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.046653 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d6d475-531b-435f-b15b-65e7ed8437b4-plugins-config-volume" pod "68d6d475-531b-435f-b15b-65e7ed8437b4" (UID: "68d6d475-531b-435f-b15b-65e7ed8437b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:23:57.046877 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.046724 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/064df158-8c5e-4cab-bef7-9b9e2a64f460-plugins-config-volume" pod "064df158-8c5e-4cab-bef7-9b9e2a64f460" (UID: "064df158-8c5e-4cab-bef7-9b9e2a64f460"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:23:57.047144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.047120 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064df158-8c5e-4cab-bef7-9b9e2a64f460-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:57.047220 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.047150 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d6d475-531b-435f-b15b-65e7ed8437b4-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:57.049076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.049051 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064df158-8c5e-4cab-bef7-9b9e2a64f460-kube-api-access-z9n89" pod "064df158-8c5e-4cab-bef7-9b9e2a64f460" (UID: "064df158-8c5e-4cab-bef7-9b9e2a64f460"). InnerVolumeSpecName "kube-api-access-z9n89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:23:57.049404 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.049384 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d6d475-531b-435f-b15b-65e7ed8437b4-kube-api-access-sk6k6" pod "68d6d475-531b-435f-b15b-65e7ed8437b4" (UID: "68d6d475-531b-435f-b15b-65e7ed8437b4"). InnerVolumeSpecName "kube-api-access-sk6k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:23:57.068014 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.067971 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:57.149704 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.149605 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9n89\" (UniqueName: \"kubernetes.io/projected/064df158-8c5e-4cab-bef7-9b9e2a64f460-kube-api-access-z9n89\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:57.149704 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.149631 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk6k6\" (UniqueName: \"kubernetes.io/projected/68d6d475-531b-435f-b15b-65e7ed8437b4-kube-api-access-sk6k6\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:23:57.292685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.292645 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:57.537659 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.537578 4413 generic.go:356] "Generic (PLEG): container finished" podID="68d6d475-531b-435f-b15b-65e7ed8437b4" containerID="4cc4b274e933585e28e753e0118d48114da528bd78c991d03037f76d90abe77f" exitCode=0 Oct 30 12:23:57.537659 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.537633 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx" Oct 30 12:23:57.537860 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.537661 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx" event={"ID":"68d6d475-531b-435f-b15b-65e7ed8437b4","Type":"ContainerDied","Data":"4cc4b274e933585e28e753e0118d48114da528bd78c991d03037f76d90abe77f"} Oct 30 12:23:57.537860 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.537722 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx" event={"ID":"68d6d475-531b-435f-b15b-65e7ed8437b4","Type":"ContainerDied","Data":"3109dabf76b285f23b5599fa5cd124a73c2634670c7da8525b9738412fada6cb"} Oct 30 12:23:57.537860 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.537737 4413 scope.go:122] "RemoveContainer" containerID="4cc4b274e933585e28e753e0118d48114da528bd78c991d03037f76d90abe77f" Oct 30 12:23:57.539324 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.539300 4413 generic.go:356] "Generic (PLEG): container finished" podID="064df158-8c5e-4cab-bef7-9b9e2a64f460" containerID="a3953db329e7e9f2493eaeec5557c3d8dba577a12c24914c5332ef6ece4fbe52" exitCode=0 Oct 30 12:23:57.539464 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.539348 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw" Oct 30 12:23:57.539464 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.539380 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw" event={"ID":"064df158-8c5e-4cab-bef7-9b9e2a64f460","Type":"ContainerDied","Data":"a3953db329e7e9f2493eaeec5557c3d8dba577a12c24914c5332ef6ece4fbe52"} Oct 30 12:23:57.539464 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.539418 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw" event={"ID":"064df158-8c5e-4cab-bef7-9b9e2a64f460","Type":"ContainerDied","Data":"5a9fb170ec334be5733c4b1e04a42e9cf65f7a1756c4195dab1c7a227357e693"} Oct 30 12:23:57.548676 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.548651 4413 scope.go:122] "RemoveContainer" containerID="4cc4b274e933585e28e753e0118d48114da528bd78c991d03037f76d90abe77f" Oct 30 12:23:57.549056 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:23:57.549036 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cc4b274e933585e28e753e0118d48114da528bd78c991d03037f76d90abe77f\": container with ID starting with 4cc4b274e933585e28e753e0118d48114da528bd78c991d03037f76d90abe77f not found: ID does not exist" containerID="4cc4b274e933585e28e753e0118d48114da528bd78c991d03037f76d90abe77f" Oct 30 12:23:57.549141 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.549069 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc4b274e933585e28e753e0118d48114da528bd78c991d03037f76d90abe77f"} err="failed to get container status \"4cc4b274e933585e28e753e0118d48114da528bd78c991d03037f76d90abe77f\": rpc error: code = NotFound desc = could not find container \"4cc4b274e933585e28e753e0118d48114da528bd78c991d03037f76d90abe77f\": container with ID starting with 4cc4b274e933585e28e753e0118d48114da528bd78c991d03037f76d90abe77f not found: ID does not exist" Oct 30 12:23:57.549141 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.549092 4413 scope.go:122] "RemoveContainer" containerID="a3953db329e7e9f2493eaeec5557c3d8dba577a12c24914c5332ef6ece4fbe52" Oct 30 12:23:57.558180 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.558163 4413 scope.go:122] "RemoveContainer" containerID="a3953db329e7e9f2493eaeec5557c3d8dba577a12c24914c5332ef6ece4fbe52" Oct 30 12:23:57.558468 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:23:57.558452 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3953db329e7e9f2493eaeec5557c3d8dba577a12c24914c5332ef6ece4fbe52\": container with ID starting with a3953db329e7e9f2493eaeec5557c3d8dba577a12c24914c5332ef6ece4fbe52 not found: ID does not exist" containerID="a3953db329e7e9f2493eaeec5557c3d8dba577a12c24914c5332ef6ece4fbe52" Oct 30 12:23:57.558544 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.558473 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3953db329e7e9f2493eaeec5557c3d8dba577a12c24914c5332ef6ece4fbe52"} err="failed to get container status \"a3953db329e7e9f2493eaeec5557c3d8dba577a12c24914c5332ef6ece4fbe52\": rpc error: code = NotFound desc = could not find container \"a3953db329e7e9f2493eaeec5557c3d8dba577a12c24914c5332ef6ece4fbe52\": container with ID starting with a3953db329e7e9f2493eaeec5557c3d8dba577a12c24914c5332ef6ece4fbe52 not found: ID does not exist" Oct 30 12:23:57.588289 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.588267 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx"] Oct 30 12:23:57.600315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.600294 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c35d6b-74886576d-96flx"] Oct 30 12:23:57.632894 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.632865 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw"] Oct 30 12:23:57.640086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.640057 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c35d6b-6998f454b7-th9kw"] Oct 30 12:23:57.664375 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.664352 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:57.817392 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:57.817235 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:58.838459 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:58.838429 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:23:59.021082 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:59.021050 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="064df158-8c5e-4cab-bef7-9b9e2a64f460" path="/var/lib/kubelet/pods/064df158-8c5e-4cab-bef7-9b9e2a64f460/volumes" Oct 30 12:23:59.021563 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:59.021541 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d6d475-531b-435f-b15b-65e7ed8437b4" path="/var/lib/kubelet/pods/68d6d475-531b-435f-b15b-65e7ed8437b4/volumes" Oct 30 12:23:59.186338 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:23:59.186303 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:00.840432 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:00.840396 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:02.538385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:02.538354 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:04.214275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:04.213798 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:04.337030 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:04.336995 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:04.573585 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:04.573481 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:05.464371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:05.464333 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:05.587800 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:05.587764 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:05.794702 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:05.794611 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr"] Oct 30 12:24:05.794873 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:05.794837 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" podUID="11ec3f83-7093-4bf6-a70e-f6eab36eac9e" containerName="istio-proxy" containerID="cri-o://29dda7754af5fae597fc14f4c2ed1c939c465fd0e5b210968e86401af155684c" gracePeriod=30 Oct 30 12:24:05.824782 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:24:05.824746 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:24:05.833984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:05.833953 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:06.318467 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:06.318432 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:06.670572 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:06.670531 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:07.069668 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.069585 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz"] Oct 30 12:24:07.070685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.070646 4413 cpu_manager.go:414] "containerMap: removing container" podUID="68d6d475-531b-435f-b15b-65e7ed8437b4" containerName="payload-processing" Oct 30 12:24:07.070685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.070674 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d6d475-531b-435f-b15b-65e7ed8437b4" containerName="payload-processing" Oct 30 12:24:07.070882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.070694 4413 cpu_manager.go:414] "containerMap: removing container" podUID="b401ad16-659d-472c-9934-26785d401eff" containerName="payload-processing" Oct 30 12:24:07.070882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.070703 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b401ad16-659d-472c-9934-26785d401eff" containerName="payload-processing" Oct 30 12:24:07.070882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.070749 4413 cpu_manager.go:414] "containerMap: removing container" podUID="86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183" containerName="main" Oct 30 12:24:07.070882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.070757 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183" containerName="main" Oct 30 12:24:07.070882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.070790 4413 cpu_manager.go:414] "containerMap: removing container" podUID="064df158-8c5e-4cab-bef7-9b9e2a64f460" containerName="payload-pre-processing" Oct 30 12:24:07.070882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.070797 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="064df158-8c5e-4cab-bef7-9b9e2a64f460" containerName="payload-pre-processing" Oct 30 12:24:07.070882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.070812 4413 cpu_manager.go:414] "containerMap: removing container" podUID="cb6b5578-4eac-409a-a8fb-f174ad8aeeb2" containerName="maas-api" Oct 30 12:24:07.070882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.070818 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb6b5578-4eac-409a-a8fb-f174ad8aeeb2" containerName="maas-api" Oct 30 12:24:07.071285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.070986 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="b401ad16-659d-472c-9934-26785d401eff" containerName="payload-processing" Oct 30 12:24:07.071285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.071009 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a3a7f8-9d16-47d3-bb8d-15f4b4d4b183" containerName="main" Oct 30 12:24:07.071285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.071023 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="064df158-8c5e-4cab-bef7-9b9e2a64f460" containerName="payload-pre-processing" Oct 30 12:24:07.071285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.071036 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb6b5578-4eac-409a-a8fb-f174ad8aeeb2" containerName="maas-api" Oct 30 12:24:07.071285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.071051 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d6d475-531b-435f-b15b-65e7ed8437b4" containerName="payload-processing" Oct 30 12:24:07.074550 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.074527 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.090291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.090225 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-922643-openshift-default-dockercfg-xswz5\"" Oct 30 12:24:07.105355 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.105321 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz"] Oct 30 12:24:07.129429 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.129390 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:07.148972 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.148937 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-workload-certs\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.149148 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.149067 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-envoy\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.149148 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.149112 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-token\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.149303 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.149195 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbfdp\" (UniqueName: \"kubernetes.io/projected/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-kube-api-access-kbfdp\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.149480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.149453 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istiod-ca-cert\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.149607 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.149583 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-data\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.149801 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.149745 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-workload-socket\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.149900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.149814 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-podinfo\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.149963 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.149935 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-credential-socket\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.296073 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.296019 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-workload-certs\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.296422 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.296379 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-envoy\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.296598 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.296580 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-token\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.296752 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.296737 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbfdp\" (UniqueName: \"kubernetes.io/projected/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-kube-api-access-kbfdp\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.296981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.296963 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istiod-ca-cert\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.297217 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.297127 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-data\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.297481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.297459 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-workload-socket\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.297611 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.297596 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-podinfo\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.297812 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.297795 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-credential-socket\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.297990 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.297948 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istiod-ca-cert\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.297990 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.296817 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-workload-certs\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.298639 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.298608 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-workload-socket\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.298912 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.298852 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-data\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.299095 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.299075 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-credential-socket\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.304710 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.304641 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-podinfo\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.307868 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.307844 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-envoy\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.325873 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.325739 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbfdp\" (UniqueName: \"kubernetes.io/projected/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-kube-api-access-kbfdp\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.331591 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.331563 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-token\") pod \"e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.393269 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.393130 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:07.500171 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.498869 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:07.733748 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.733723 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz"] Oct 30 12:24:07.735649 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:24:07.735620 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67aa5abb_dd1e_4ac9_888e_bb958fddffa9.slice/crio-a4e97fddc13ec7a61a943824c8b14ca9978f2f4767b71e388dfc1f4770f9f88b WatchSource:0}: Error finding container a4e97fddc13ec7a61a943824c8b14ca9978f2f4767b71e388dfc1f4770f9f88b: Status 404 returned error can't find the container with id a4e97fddc13ec7a61a943824c8b14ca9978f2f4767b71e388dfc1f4770f9f88b Oct 30 12:24:07.738776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.738734 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:24:07.738905 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.738829 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:24:07.738905 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.738878 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:24:07.952350 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.952311 4413 patch_prober.go:29] interesting pod/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:24:07.952508 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:07.952382 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" podUID="11ec3f83-7093-4bf6-a70e-f6eab36eac9e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:24:08.116960 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:08.116854 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:08.617461 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:08.617427 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:08.626626 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:08.626591 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" event={"ID":"67aa5abb-dd1e-4ac9-888e-bb958fddffa9","Type":"ContainerStarted","Data":"06dba9e671558e66e3ce60482f497730e5a6067fdf2cf9a225ea42e53aa2f5e4"} Oct 30 12:24:08.626802 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:08.626676 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" event={"ID":"67aa5abb-dd1e-4ac9-888e-bb958fddffa9","Type":"ContainerStarted","Data":"a4e97fddc13ec7a61a943824c8b14ca9978f2f4767b71e388dfc1f4770f9f88b"} Oct 30 12:24:08.682566 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:08.682515 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" podStartSLOduration=1.682502666 podStartE2EDuration="1.682502666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:08.681868065 +0000 UTC m=+1801.791710946" watchObservedRunningTime="2026-10-30 12:24:08.682502666 +0000 UTC m=+1801.792345585" Oct 30 12:24:09.082041 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:09.081997 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:09.393797 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:09.393763 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:09.400852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:09.400828 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:09.633559 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:09.633518 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:09.634707 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:09.634687 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:09.648636 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:09.648548 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:09.762977 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:09.762942 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:10.503508 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:10.503480 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:10.621814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:10.621774 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:10.735510 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:10.735478 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:11.077696 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.077617 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:24:11.137392 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.137351 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:11.160387 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.160351 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-data\") pod \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " Oct 30 12:24:11.160574 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.160403 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-podinfo\") pod \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " Oct 30 12:24:11.160574 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.160452 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-credential-socket\") pod \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " Oct 30 12:24:11.160701 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.160630 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istiod-ca-cert\") pod \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " Oct 30 12:24:11.160701 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.160683 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-data" pod "11ec3f83-7093-4bf6-a70e-f6eab36eac9e" (UID: "11ec3f83-7093-4bf6-a70e-f6eab36eac9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:11.160816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.160704 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-workload-certs\") pod \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " Oct 30 12:24:11.160816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.160754 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-kube-api-access-g5w52\" (UniqueName: \"kubernetes.io/projected/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-kube-api-access-g5w52\") pod \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " Oct 30 12:24:11.160816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.160784 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-workload-socket\") pod \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " Oct 30 12:24:11.160974 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.160828 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-envoy\") pod \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " Oct 30 12:24:11.160974 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.160873 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-token\" (UniqueName: \"kubernetes.io/projected/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-token\") pod \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\" (UID: \"11ec3f83-7093-4bf6-a70e-f6eab36eac9e\") " Oct 30 12:24:11.161073 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.160825 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-credential-socket" pod "11ec3f83-7093-4bf6-a70e-f6eab36eac9e" (UID: "11ec3f83-7093-4bf6-a70e-f6eab36eac9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:11.161073 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.161043 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-workload-socket" pod "11ec3f83-7093-4bf6-a70e-f6eab36eac9e" (UID: "11ec3f83-7093-4bf6-a70e-f6eab36eac9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:11.161073 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.161026 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istiod-ca-cert" pod "11ec3f83-7093-4bf6-a70e-f6eab36eac9e" (UID: "11ec3f83-7093-4bf6-a70e-f6eab36eac9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:24:11.161765 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.161664 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.161898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.161858 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.161898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.161875 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.161898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.161890 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.162062 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.161707 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-workload-certs" pod "11ec3f83-7093-4bf6-a70e-f6eab36eac9e" (UID: "11ec3f83-7093-4bf6-a70e-f6eab36eac9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:11.163100 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.163074 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-podinfo" pod "11ec3f83-7093-4bf6-a70e-f6eab36eac9e" (UID: "11ec3f83-7093-4bf6-a70e-f6eab36eac9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:24:11.163418 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.163376 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-token" pod "11ec3f83-7093-4bf6-a70e-f6eab36eac9e" (UID: "11ec3f83-7093-4bf6-a70e-f6eab36eac9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:24:11.163516 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.163470 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-envoy" pod "11ec3f83-7093-4bf6-a70e-f6eab36eac9e" (UID: "11ec3f83-7093-4bf6-a70e-f6eab36eac9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:24:11.163960 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.163941 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-kube-api-access-g5w52" pod "11ec3f83-7093-4bf6-a70e-f6eab36eac9e" (UID: "11ec3f83-7093-4bf6-a70e-f6eab36eac9e"). InnerVolumeSpecName "kube-api-access-g5w52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:24:11.259441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.259399 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:11.263553 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.263523 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.263553 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.263555 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5w52\" (UniqueName: \"kubernetes.io/projected/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-kube-api-access-g5w52\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.263852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.263572 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.263852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.263582 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.263852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.263590 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ec3f83-7093-4bf6-a70e-f6eab36eac9e-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:24:11.649946 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.649913 4413 generic.go:356] "Generic (PLEG): container finished" podID="11ec3f83-7093-4bf6-a70e-f6eab36eac9e" containerID="29dda7754af5fae597fc14f4c2ed1c939c465fd0e5b210968e86401af155684c" exitCode=0 Oct 30 12:24:11.650436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.649963 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" Oct 30 12:24:11.650436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.649992 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" event={"ID":"11ec3f83-7093-4bf6-a70e-f6eab36eac9e","Type":"ContainerDied","Data":"29dda7754af5fae597fc14f4c2ed1c939c465fd0e5b210968e86401af155684c"} Oct 30 12:24:11.650436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.650031 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr" event={"ID":"11ec3f83-7093-4bf6-a70e-f6eab36eac9e","Type":"ContainerDied","Data":"348d4d4b8dc12fa327a19af2a62292b31af4835034808f15567cb96a92ab3499"} Oct 30 12:24:11.650436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.650052 4413 scope.go:122] "RemoveContainer" containerID="29dda7754af5fae597fc14f4c2ed1c939c465fd0e5b210968e86401af155684c" Oct 30 12:24:11.662391 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.662362 4413 scope.go:122] "RemoveContainer" containerID="06c3e1f4b9ccd66d35f5f81a61e14ed85286638d68edbe6e576478df61e90598" Oct 30 12:24:11.673637 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.673613 4413 scope.go:122] "RemoveContainer" containerID="29dda7754af5fae597fc14f4c2ed1c939c465fd0e5b210968e86401af155684c" Oct 30 12:24:11.673962 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:24:11.673942 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29dda7754af5fae597fc14f4c2ed1c939c465fd0e5b210968e86401af155684c\": container with ID starting with 29dda7754af5fae597fc14f4c2ed1c939c465fd0e5b210968e86401af155684c not found: ID does not exist" containerID="29dda7754af5fae597fc14f4c2ed1c939c465fd0e5b210968e86401af155684c" Oct 30 12:24:11.674047 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.673973 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29dda7754af5fae597fc14f4c2ed1c939c465fd0e5b210968e86401af155684c"} err="failed to get container status \"29dda7754af5fae597fc14f4c2ed1c939c465fd0e5b210968e86401af155684c\": rpc error: code = NotFound desc = could not find container \"29dda7754af5fae597fc14f4c2ed1c939c465fd0e5b210968e86401af155684c\": container with ID starting with 29dda7754af5fae597fc14f4c2ed1c939c465fd0e5b210968e86401af155684c not found: ID does not exist" Oct 30 12:24:11.674047 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.673997 4413 scope.go:122] "RemoveContainer" containerID="06c3e1f4b9ccd66d35f5f81a61e14ed85286638d68edbe6e576478df61e90598" Oct 30 12:24:11.674319 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:24:11.674298 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c3e1f4b9ccd66d35f5f81a61e14ed85286638d68edbe6e576478df61e90598\": container with ID starting with 06c3e1f4b9ccd66d35f5f81a61e14ed85286638d68edbe6e576478df61e90598 not found: ID does not exist" containerID="06c3e1f4b9ccd66d35f5f81a61e14ed85286638d68edbe6e576478df61e90598" Oct 30 12:24:11.674356 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.674326 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c3e1f4b9ccd66d35f5f81a61e14ed85286638d68edbe6e576478df61e90598"} err="failed to get container status \"06c3e1f4b9ccd66d35f5f81a61e14ed85286638d68edbe6e576478df61e90598\": rpc error: code = NotFound desc = could not find container \"06c3e1f4b9ccd66d35f5f81a61e14ed85286638d68edbe6e576478df61e90598\": container with ID starting with 06c3e1f4b9ccd66d35f5f81a61e14ed85286638d68edbe6e576478df61e90598 not found: ID does not exist" Oct 30 12:24:11.718888 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.718858 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr"] Oct 30 12:24:11.722094 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:11.722072 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-060f41f2-openshift-default-5f86797c8f-2wvhr"] Oct 30 12:24:12.739696 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:12.739656 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:12.967821 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:12.967786 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:13.026495 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.026410 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ec3f83-7093-4bf6-a70e-f6eab36eac9e" path="/var/lib/kubelet/pods/11ec3f83-7093-4bf6-a70e-f6eab36eac9e/volumes" Oct 30 12:24:13.081152 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.081116 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:13.132949 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.132920 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm"] Oct 30 12:24:13.133562 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.133548 4413 cpu_manager.go:414] "containerMap: removing container" podUID="11ec3f83-7093-4bf6-a70e-f6eab36eac9e" containerName="istio-proxy" Oct 30 12:24:13.133639 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.133564 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ec3f83-7093-4bf6-a70e-f6eab36eac9e" containerName="istio-proxy" Oct 30 12:24:13.133688 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.133680 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ec3f83-7093-4bf6-a70e-f6eab36eac9e" containerName="istio-proxy" Oct 30 12:24:13.133724 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.133699 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ec3f83-7093-4bf6-a70e-f6eab36eac9e" containerName="istio-proxy" Oct 30 12:24:13.137198 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.137180 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm" Oct 30 12:24:13.154885 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.154858 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm"] Oct 30 12:24:13.163078 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.163056 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-922643\"" Oct 30 12:24:13.163208 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.163151 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-922643-dockercfg-ckztw\"" Oct 30 12:24:13.194223 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.194189 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3369323e-156f-44da-b214-6d8e63a97956-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm\" (UID: \"3369323e-156f-44da-b214-6d8e63a97956\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm" Oct 30 12:24:13.194480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.194456 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvjzw\" (UniqueName: \"kubernetes.io/projected/3369323e-156f-44da-b214-6d8e63a97956-kube-api-access-xvjzw\") pod \"payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm\" (UID: \"3369323e-156f-44da-b214-6d8e63a97956\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm" Oct 30 12:24:13.306984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.306886 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvjzw\" (UniqueName: \"kubernetes.io/projected/3369323e-156f-44da-b214-6d8e63a97956-kube-api-access-xvjzw\") pod \"payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm\" (UID: \"3369323e-156f-44da-b214-6d8e63a97956\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm" Oct 30 12:24:13.307446 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.307420 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3369323e-156f-44da-b214-6d8e63a97956-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm\" (UID: \"3369323e-156f-44da-b214-6d8e63a97956\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm" Oct 30 12:24:13.324921 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.308640 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3369323e-156f-44da-b214-6d8e63a97956-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm\" (UID: \"3369323e-156f-44da-b214-6d8e63a97956\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm" Oct 30 12:24:13.327659 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.327175 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g"] Oct 30 12:24:13.331629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.329014 4413 cpu_manager.go:414] "containerMap: removing container" podUID="11ec3f83-7093-4bf6-a70e-f6eab36eac9e" containerName="istio-proxy" Oct 30 12:24:13.331817 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.331798 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ec3f83-7093-4bf6-a70e-f6eab36eac9e" containerName="istio-proxy" Oct 30 12:24:13.337325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.337297 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g"] Oct 30 12:24:13.337513 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.337486 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g" Oct 30 12:24:13.351318 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.351284 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvjzw\" (UniqueName: \"kubernetes.io/projected/3369323e-156f-44da-b214-6d8e63a97956-kube-api-access-xvjzw\") pod \"payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm\" (UID: \"3369323e-156f-44da-b214-6d8e63a97956\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm" Oct 30 12:24:13.451743 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.451716 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm" Oct 30 12:24:13.516083 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.514236 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g\" (UID: \"29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe\") " pod="openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g" Oct 30 12:24:13.516083 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.514627 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxghs\" (UniqueName: \"kubernetes.io/projected/29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe-kube-api-access-qxghs\") pod \"payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g\" (UID: \"29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe\") " pod="openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g" Oct 30 12:24:13.615955 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.615866 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g\" (UID: \"29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe\") " pod="openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g" Oct 30 12:24:13.616179 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.616162 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxghs\" (UniqueName: \"kubernetes.io/projected/29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe-kube-api-access-qxghs\") pod \"payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g\" (UID: \"29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe\") " pod="openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g" Oct 30 12:24:13.616695 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.616676 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g\" (UID: \"29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe\") " pod="openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g" Oct 30 12:24:13.639184 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.639155 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxghs\" (UniqueName: \"kubernetes.io/projected/29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe-kube-api-access-qxghs\") pod \"payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g\" (UID: \"29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe\") " pod="openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g" Oct 30 12:24:13.682898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.682869 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g" Oct 30 12:24:13.767913 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.767874 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm"] Oct 30 12:24:13.774467 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:24:13.774435 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3369323e_156f_44da_b214_6d8e63a97956.slice/crio-2135df74a5ec392d967ce9b4f328c777ec64c0253962ede44ebfbc276efa94c7 WatchSource:0}: Error finding container 2135df74a5ec392d967ce9b4f328c777ec64c0253962ede44ebfbc276efa94c7: Status 404 returned error can't find the container with id 2135df74a5ec392d967ce9b4f328c777ec64c0253962ede44ebfbc276efa94c7 Oct 30 12:24:13.898857 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.898802 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:13.986511 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:13.986483 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g"] Oct 30 12:24:13.987909 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:24:13.987881 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29aaf5f4_e6f6_4b96_8c71_8b13a2cacebe.slice/crio-1a0f24bc7907f50efb2ba9f57faac25a060d3f58e046107f2388c508003ea893 WatchSource:0}: Error finding container 1a0f24bc7907f50efb2ba9f57faac25a060d3f58e046107f2388c508003ea893: Status 404 returned error can't find the container with id 1a0f24bc7907f50efb2ba9f57faac25a060d3f58e046107f2388c508003ea893 Oct 30 12:24:14.371142 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:14.371112 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:14.543232 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:14.543203 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:14.657425 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:14.657392 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:14.673886 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:14.673854 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm" event={"ID":"3369323e-156f-44da-b214-6d8e63a97956","Type":"ContainerStarted","Data":"4db8aae06533eddafb1f4fa529e1cb1a7da6661575669fa1f6b7802bd7a065cb"} Oct 30 12:24:14.674086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:14.673914 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm" event={"ID":"3369323e-156f-44da-b214-6d8e63a97956","Type":"ContainerStarted","Data":"2135df74a5ec392d967ce9b4f328c777ec64c0253962ede44ebfbc276efa94c7"} Oct 30 12:24:14.674086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:14.673940 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm" Oct 30 12:24:14.675931 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:14.675910 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g" event={"ID":"29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe","Type":"ContainerStarted","Data":"cdc1196e8caca77d4b06db0c8d0095f9fa73c0402b9594086450f8d9530ea211"} Oct 30 12:24:14.676065 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:14.675949 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g" event={"ID":"29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe","Type":"ContainerStarted","Data":"1a0f24bc7907f50efb2ba9f57faac25a060d3f58e046107f2388c508003ea893"} Oct 30 12:24:14.676065 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:14.676002 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g" Oct 30 12:24:14.719606 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:14.719549 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm" podStartSLOduration=1.719532619 podStartE2EDuration="1.719532619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:14.718889013 +0000 UTC m=+1807.828731932" watchObservedRunningTime="2026-10-30 12:24:14.719532619 +0000 UTC m=+1807.829375511" Oct 30 12:24:14.760834 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:14.760783 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g" podStartSLOduration=1.760767082 podStartE2EDuration="1.760767082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:14.75746746 +0000 UTC m=+1807.867310340" watchObservedRunningTime="2026-10-30 12:24:14.760767082 +0000 UTC m=+1807.870609999" Oct 30 12:24:16.349824 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:16.349792 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:16.858232 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:16.858201 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:16.969010 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:16.968977 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:17.487571 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:17.487533 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:17.615273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:17.614850 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:17.817599 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:17.817518 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:18.302569 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:18.302539 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:19.360019 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:19.359981 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:20.101120 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:20.101087 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:21.753717 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:21.753687 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:23.326061 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:23.325969 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6"] Oct 30 12:24:23.329641 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:23.329617 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:24:23.350194 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:23.350169 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6"] Oct 30 12:24:23.357681 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:23.357639 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-922643-serving-cert\"" Oct 30 12:24:23.413662 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:23.413627 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:23.467723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:23.467684 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smtd9\" (UniqueName: \"kubernetes.io/projected/c842dc5f-e6cc-4133-bd37-6c95f466adca-kube-api-access-smtd9\") pod \"maas-api-e2e-api-a-922643-69c598bf54-2cgg6\" (UID: \"c842dc5f-e6cc-4133-bd37-6c95f466adca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:24:23.467906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:23.467888 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c842dc5f-e6cc-4133-bd37-6c95f466adca-maas-api-tls\") pod \"maas-api-e2e-api-a-922643-69c598bf54-2cgg6\" (UID: \"c842dc5f-e6cc-4133-bd37-6c95f466adca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:24:23.467994 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:23.467983 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c842dc5f-e6cc-4133-bd37-6c95f466adca-metrics-certs\") pod \"maas-api-e2e-api-a-922643-69c598bf54-2cgg6\" (UID: \"c842dc5f-e6cc-4133-bd37-6c95f466adca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:24:23.468115 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:23.468089 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c842dc5f-e6cc-4133-bd37-6c95f466adca-openshift-service-ca\") pod \"maas-api-e2e-api-a-922643-69c598bf54-2cgg6\" (UID: \"c842dc5f-e6cc-4133-bd37-6c95f466adca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:24:23.524948 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:23.524905 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:23.569063 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:23.569034 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c842dc5f-e6cc-4133-bd37-6c95f466adca-maas-api-tls\") pod \"maas-api-e2e-api-a-922643-69c598bf54-2cgg6\" (UID: \"c842dc5f-e6cc-4133-bd37-6c95f466adca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:24:23.569347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:23.569143 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c842dc5f-e6cc-4133-bd37-6c95f466adca-metrics-certs\") pod \"maas-api-e2e-api-a-922643-69c598bf54-2cgg6\" (UID: \"c842dc5f-e6cc-4133-bd37-6c95f466adca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:24:23.569347 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:24:23.569212 4413 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-922643-serving-cert: secret "maas-api-e2e-api-a-922643-serving-cert" not found Oct 30 12:24:23.569347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:23.569227 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c842dc5f-e6cc-4133-bd37-6c95f466adca-openshift-service-ca\") pod \"maas-api-e2e-api-a-922643-69c598bf54-2cgg6\" (UID: \"c842dc5f-e6cc-4133-bd37-6c95f466adca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:24:23.569347 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:24:23.569301 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c842dc5f-e6cc-4133-bd37-6c95f466adca-maas-api-tls podName:c842dc5f-e6cc-4133-bd37-6c95f466adca nodeName:}" failed. No retries permitted until 2026-10-30 12:24:24.069279498 +0000 UTC m=+1817.179122374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c842dc5f-e6cc-4133-bd37-6c95f466adca-maas-api-tls") pod "maas-api-e2e-api-a-922643-69c598bf54-2cgg6" (UID: "c842dc5f-e6cc-4133-bd37-6c95f466adca") : secret "maas-api-e2e-api-a-922643-serving-cert" not found Oct 30 12:24:23.569606 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:23.569367 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smtd9\" (UniqueName: \"kubernetes.io/projected/c842dc5f-e6cc-4133-bd37-6c95f466adca-kube-api-access-smtd9\") pod \"maas-api-e2e-api-a-922643-69c598bf54-2cgg6\" (UID: \"c842dc5f-e6cc-4133-bd37-6c95f466adca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:24:23.570093 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:23.570063 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c842dc5f-e6cc-4133-bd37-6c95f466adca-openshift-service-ca\") pod \"maas-api-e2e-api-a-922643-69c598bf54-2cgg6\" (UID: \"c842dc5f-e6cc-4133-bd37-6c95f466adca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:24:23.571963 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:23.571939 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c842dc5f-e6cc-4133-bd37-6c95f466adca-metrics-certs\") pod \"maas-api-e2e-api-a-922643-69c598bf54-2cgg6\" (UID: \"c842dc5f-e6cc-4133-bd37-6c95f466adca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:24:23.592285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:23.592200 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smtd9\" (UniqueName: \"kubernetes.io/projected/c842dc5f-e6cc-4133-bd37-6c95f466adca-kube-api-access-smtd9\") pod \"maas-api-e2e-api-a-922643-69c598bf54-2cgg6\" (UID: \"c842dc5f-e6cc-4133-bd37-6c95f466adca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:24:23.693244 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:23.693206 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:24.076764 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:24.076727 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c842dc5f-e6cc-4133-bd37-6c95f466adca-maas-api-tls\") pod \"maas-api-e2e-api-a-922643-69c598bf54-2cgg6\" (UID: \"c842dc5f-e6cc-4133-bd37-6c95f466adca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:24:24.079233 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:24.079210 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c842dc5f-e6cc-4133-bd37-6c95f466adca-maas-api-tls\") pod \"maas-api-e2e-api-a-922643-69c598bf54-2cgg6\" (UID: \"c842dc5f-e6cc-4133-bd37-6c95f466adca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:24:24.243608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:24.243575 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:24:24.578361 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:24.578321 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6"] Oct 30 12:24:24.584296 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:24:24.584224 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc842dc5f_e6cc_4133_bd37_6c95f466adca.slice/crio-b7e7af5481fc69b3ad4b71a9c5649f608fc6a7dac642ad4786be44d4f7ab267e WatchSource:0}: Error finding container b7e7af5481fc69b3ad4b71a9c5649f608fc6a7dac642ad4786be44d4f7ab267e: Status 404 returned error can't find the container with id b7e7af5481fc69b3ad4b71a9c5649f608fc6a7dac642ad4786be44d4f7ab267e Oct 30 12:24:24.771586 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:24.771549 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" event={"ID":"c842dc5f-e6cc-4133-bd37-6c95f466adca","Type":"ContainerStarted","Data":"b7e7af5481fc69b3ad4b71a9c5649f608fc6a7dac642ad4786be44d4f7ab267e"} Oct 30 12:24:24.833643 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:24.833567 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:25.241566 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:25.241534 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:25.684764 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:25.684738 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g" Oct 30 12:24:25.685230 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:25.684822 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm" Oct 30 12:24:25.784792 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:25.784759 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" event={"ID":"c842dc5f-e6cc-4133-bd37-6c95f466adca","Type":"ContainerStarted","Data":"d07bebaef464ef602373c0ffa4fc7568b2af74e084d74e9daa34e64f7de748f4"} Oct 30 12:24:25.785074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:25.785052 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:24:25.828376 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:25.828327 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" podStartSLOduration=2.378828899 podStartE2EDuration="2.828309075s" totalImagesPullingTime="449.480176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:25.826535602 +0000 UTC m=+1818.936378485" watchObservedRunningTime="2026-10-30 12:24:25.828309075 +0000 UTC m=+1818.938151956" Oct 30 12:24:26.983185 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:26.983146 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:28.619167 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:28.619133 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:28.990472 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:28.990442 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:29.108760 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:29.108724 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:30.339373 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:30.339339 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:31.797212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:31.797179 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:24:31.921239 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:31.921202 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:33.559239 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:33.559204 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:34.450912 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:34.450862 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:34.892325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:34.892282 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f"] Oct 30 12:24:34.896811 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:34.896792 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:34.914661 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:34.914637 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-951ad1-openshift-default-dockercfg-88nnd\"" Oct 30 12:24:34.927415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:34.927391 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f"] Oct 30 12:24:34.944570 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:34.944540 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:34.975626 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:34.975585 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-token\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:34.975794 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:34.975651 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-workload-socket\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:34.975794 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:34.975694 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dd9r\" (UniqueName: \"kubernetes.io/projected/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-kube-api-access-5dd9r\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:34.975906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:34.975817 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-envoy\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:34.975906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:34.975865 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-workload-certs\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:34.976000 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:34.975951 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-data\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:34.976155 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:34.976107 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istiod-ca-cert\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:34.976375 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:34.976350 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-podinfo\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:34.976561 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:34.976543 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-credential-socket\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.078237 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.078194 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-podinfo\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.099406 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.099371 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-credential-socket\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.109301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.099852 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-token\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.109555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.109531 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-workload-socket\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.131398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.131364 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dd9r\" (UniqueName: \"kubernetes.io/projected/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-kube-api-access-5dd9r\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.131674 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.131653 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-envoy\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.134086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.101313 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-credential-socket\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.134086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.131802 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-workload-certs\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.134086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.131876 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-data\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.134086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.132067 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istiod-ca-cert\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.134086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.132129 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-workload-certs\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.134086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.132609 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-workload-socket\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.134086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.132824 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-data\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.134086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.133407 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istiod-ca-cert\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.134739 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.134718 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-podinfo\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.177633 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.177559 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dd9r\" (UniqueName: \"kubernetes.io/projected/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-kube-api-access-5dd9r\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.180847 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.179834 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-token\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.181281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.181242 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-envoy\") pod \"e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.212116 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.212088 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:35.512944 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.512858 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:35.564887 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.564857 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f"] Oct 30 12:24:35.567901 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:24:35.567874 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30dcb8a5_3b7e_4d7b_8145_996efaf9e84a.slice/crio-450268b80971897d89630c76745e5a5b8721556023c186c730b66abbf3bdf987 WatchSource:0}: Error finding container 450268b80971897d89630c76745e5a5b8721556023c186c730b66abbf3bdf987: Status 404 returned error can't find the container with id 450268b80971897d89630c76745e5a5b8721556023c186c730b66abbf3bdf987 Oct 30 12:24:35.570588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.570557 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:24:35.570652 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.570617 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:24:35.570652 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.570648 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:24:35.634285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.633320 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:35.841667 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:24:35.841582 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:24:35.853032 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.853002 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" event={"ID":"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a","Type":"ContainerStarted","Data":"fa61216a661768c2ffb1cfc935637ff581c445fce161d3b4f57521d3abf1d12f"} Oct 30 12:24:35.853203 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:35.853063 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" event={"ID":"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a","Type":"ContainerStarted","Data":"450268b80971897d89630c76745e5a5b8721556023c186c730b66abbf3bdf987"} Oct 30 12:24:36.216274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:36.215242 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:36.221594 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:36.221569 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:36.266129 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:36.266075 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" podStartSLOduration=2.26605964 podStartE2EDuration="2.26605964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:35.909703475 +0000 UTC m=+1829.019546357" watchObservedRunningTime="2026-10-30 12:24:36.26605964 +0000 UTC m=+1829.375902582" Oct 30 12:24:36.860773 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:36.860735 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:36.861804 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:36.861783 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:24:37.371612 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:37.371583 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:38.477044 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:38.476999 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:38.610827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:38.610789 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:39.308317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:39.308243 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:40.139773 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:40.139715 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:40.250550 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:40.250513 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:40.487670 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:40.477690 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx"] Oct 30 12:24:40.489390 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:40.489364 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx" Oct 30 12:24:40.506061 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:40.506024 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx"] Oct 30 12:24:40.642273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:40.638505 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-951ad1-dockercfg-8425s\"" Oct 30 12:24:40.642273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:40.638591 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-951ad1\"" Oct 30 12:24:40.642273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:40.640385 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jchd\" (UniqueName: \"kubernetes.io/projected/dfcc37aa-99af-4a3c-bf39-34d2118506e6-kube-api-access-4jchd\") pod \"payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx\" (UID: \"dfcc37aa-99af-4a3c-bf39-34d2118506e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx" Oct 30 12:24:40.642273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:40.640624 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcc37aa-99af-4a3c-bf39-34d2118506e6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx\" (UID: \"dfcc37aa-99af-4a3c-bf39-34d2118506e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx" Oct 30 12:24:40.742996 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:40.742904 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jchd\" (UniqueName: \"kubernetes.io/projected/dfcc37aa-99af-4a3c-bf39-34d2118506e6-kube-api-access-4jchd\") pod \"payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx\" (UID: \"dfcc37aa-99af-4a3c-bf39-34d2118506e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx" Oct 30 12:24:40.743166 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:40.743052 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcc37aa-99af-4a3c-bf39-34d2118506e6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx\" (UID: \"dfcc37aa-99af-4a3c-bf39-34d2118506e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx" Oct 30 12:24:40.743814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:40.743794 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcc37aa-99af-4a3c-bf39-34d2118506e6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx\" (UID: \"dfcc37aa-99af-4a3c-bf39-34d2118506e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx" Oct 30 12:24:40.766139 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:40.766105 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jchd\" (UniqueName: \"kubernetes.io/projected/dfcc37aa-99af-4a3c-bf39-34d2118506e6-kube-api-access-4jchd\") pod \"payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx\" (UID: \"dfcc37aa-99af-4a3c-bf39-34d2118506e6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx" Oct 30 12:24:40.839467 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:40.839432 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx" Oct 30 12:24:41.161499 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:41.161466 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:41.169280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:41.169241 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx"] Oct 30 12:24:41.170547 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:24:41.170524 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfcc37aa_99af_4a3c_bf39_34d2118506e6.slice/crio-fa9cbcb80508d19c4638a5694511f03be4002d52e7d65c6317d737b463fe3ee5 WatchSource:0}: Error finding container fa9cbcb80508d19c4638a5694511f03be4002d52e7d65c6317d737b463fe3ee5: Status 404 returned error can't find the container with id fa9cbcb80508d19c4638a5694511f03be4002d52e7d65c6317d737b463fe3ee5 Oct 30 12:24:41.405240 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:41.405201 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:41.896424 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:41.896387 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx" event={"ID":"dfcc37aa-99af-4a3c-bf39-34d2118506e6","Type":"ContainerStarted","Data":"2b187a83eb5fb68a5cd7c5fe99b9b160d25a39d1a74393995f52fcbc04a24fa6"} Oct 30 12:24:41.896575 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:41.896498 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx" event={"ID":"dfcc37aa-99af-4a3c-bf39-34d2118506e6","Type":"ContainerStarted","Data":"fa9cbcb80508d19c4638a5694511f03be4002d52e7d65c6317d737b463fe3ee5"} Oct 30 12:24:41.896575 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:41.896530 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx" Oct 30 12:24:41.936322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:41.936226 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx" podStartSLOduration=1.936209074 podStartE2EDuration="1.936209074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:41.935379489 +0000 UTC m=+1835.045222371" watchObservedRunningTime="2026-10-30 12:24:41.936209074 +0000 UTC m=+1835.046051956" Oct 30 12:24:42.122915 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:42.122861 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:42.735025 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:42.734969 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:42.876897 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:42.876780 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:43.010332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:43.010243 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:44.763624 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:44.763591 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:45.947694 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:45.947663 4413 generic.go:356] "Generic (PLEG): container finished" podID="67aa5abb-dd1e-4ac9-888e-bb958fddffa9" containerID="06dba9e671558e66e3ce60482f497730e5a6067fdf2cf9a225ea42e53aa2f5e4" exitCode=0 Oct 30 12:24:45.947694 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:45.947706 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" event={"ID":"67aa5abb-dd1e-4ac9-888e-bb958fddffa9","Type":"ContainerDied","Data":"06dba9e671558e66e3ce60482f497730e5a6067fdf2cf9a225ea42e53aa2f5e4"} Oct 30 12:24:45.948341 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:45.948179 4413 scope.go:122] "RemoveContainer" containerID="06dba9e671558e66e3ce60482f497730e5a6067fdf2cf9a225ea42e53aa2f5e4" Oct 30 12:24:45.949741 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:45.949708 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:24:45.949834 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:45.949780 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:24:45.949834 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:45.949813 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:24:46.609044 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:46.590580 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:46.723273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:46.717032 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:46.852280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:46.852226 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:46.956106 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:46.956072 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" event={"ID":"67aa5abb-dd1e-4ac9-888e-bb958fddffa9","Type":"ContainerStarted","Data":"762ee8d14c218d84d114435177787d40ac93425aecb31ee70027fee69949bf42"} Oct 30 12:24:47.411273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:47.397618 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:47.411273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:47.397761 4413 patch_prober.go:29] interesting pod/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 12:24:47.411273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:47.397821 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" podUID="67aa5abb-dd1e-4ac9-888e-bb958fddffa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 12:24:48.125398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.123646 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j"] Oct 30 12:24:48.125398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.123898 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" podUID="661c97b4-755e-44af-b5bb-bf67720638bb" containerName="maas-api" containerID="cri-o://a39c29425bed5b3ded1fd6da679612b782ac444a855b76fbbe528b5f7296c1df" gracePeriod=30 Oct 30 12:24:48.216460 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.216421 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:48.437360 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.400637 4413 patch_prober.go:29] interesting pod/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 12:24:48.437360 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.400703 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" podUID="67aa5abb-dd1e-4ac9-888e-bb958fddffa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 12:24:48.440413 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.440382 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:48.482324 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.482300 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:24:48.566473 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.566443 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:48.583146 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.583111 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt"] Oct 30 12:24:48.583872 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.583855 4413 cpu_manager.go:414] "containerMap: removing container" podUID="661c97b4-755e-44af-b5bb-bf67720638bb" containerName="maas-api" Oct 30 12:24:48.583922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.583875 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661c97b4-755e-44af-b5bb-bf67720638bb" containerName="maas-api" Oct 30 12:24:48.583991 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.583981 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="661c97b4-755e-44af-b5bb-bf67720638bb" containerName="maas-api" Oct 30 12:24:48.586468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.586444 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt" Oct 30 12:24:48.621896 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.620644 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-aef61e\"" Oct 30 12:24:48.621896 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.620818 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-aef61e-dockercfg-p4rnp\"" Oct 30 12:24:48.630289 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.629599 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt"] Oct 30 12:24:48.649168 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.647874 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661c97b4-755e-44af-b5bb-bf67720638bb-kube-api-access-7p8cj\" (UniqueName: \"kubernetes.io/projected/661c97b4-755e-44af-b5bb-bf67720638bb-kube-api-access-7p8cj\") pod \"661c97b4-755e-44af-b5bb-bf67720638bb\" (UID: \"661c97b4-755e-44af-b5bb-bf67720638bb\") " Oct 30 12:24:48.649168 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.648039 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/661c97b4-755e-44af-b5bb-bf67720638bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/661c97b4-755e-44af-b5bb-bf67720638bb-metrics-certs\") pod \"661c97b4-755e-44af-b5bb-bf67720638bb\" (UID: \"661c97b4-755e-44af-b5bb-bf67720638bb\") " Oct 30 12:24:48.649168 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.648628 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/661c97b4-755e-44af-b5bb-bf67720638bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661c97b4-755e-44af-b5bb-bf67720638bb-openshift-service-ca\") pod \"661c97b4-755e-44af-b5bb-bf67720638bb\" (UID: \"661c97b4-755e-44af-b5bb-bf67720638bb\") " Oct 30 12:24:48.649168 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.648709 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/661c97b4-755e-44af-b5bb-bf67720638bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661c97b4-755e-44af-b5bb-bf67720638bb-maas-api-tls\") pod \"661c97b4-755e-44af-b5bb-bf67720638bb\" (UID: \"661c97b4-755e-44af-b5bb-bf67720638bb\") " Oct 30 12:24:48.650795 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.650701 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661c97b4-755e-44af-b5bb-bf67720638bb-openshift-service-ca" pod "661c97b4-755e-44af-b5bb-bf67720638bb" (UID: "661c97b4-755e-44af-b5bb-bf67720638bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:24:48.662000 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.654411 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661c97b4-755e-44af-b5bb-bf67720638bb-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:24:48.855723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.854312 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:48.858491 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.858465 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbq5l\" (UniqueName: \"kubernetes.io/projected/a488bb31-f27f-4d15-85a1-f11468ea3fec-kube-api-access-wbq5l\") pod \"payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt\" (UID: \"a488bb31-f27f-4d15-85a1-f11468ea3fec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt" Oct 30 12:24:48.858800 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.858782 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a488bb31-f27f-4d15-85a1-f11468ea3fec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt\" (UID: \"a488bb31-f27f-4d15-85a1-f11468ea3fec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt" Oct 30 12:24:48.939374 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.913971 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/661c97b4-755e-44af-b5bb-bf67720638bb-metrics-certs" pod "661c97b4-755e-44af-b5bb-bf67720638bb" (UID: "661c97b4-755e-44af-b5bb-bf67720638bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:24:48.946410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.946361 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661c97b4-755e-44af-b5bb-bf67720638bb-kube-api-access-7p8cj" pod "661c97b4-755e-44af-b5bb-bf67720638bb" (UID: "661c97b4-755e-44af-b5bb-bf67720638bb"). InnerVolumeSpecName "kube-api-access-7p8cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:24:48.950826 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.950769 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/661c97b4-755e-44af-b5bb-bf67720638bb-maas-api-tls" pod "661c97b4-755e-44af-b5bb-bf67720638bb" (UID: "661c97b4-755e-44af-b5bb-bf67720638bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:24:48.960914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.960855 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbq5l\" (UniqueName: \"kubernetes.io/projected/a488bb31-f27f-4d15-85a1-f11468ea3fec-kube-api-access-wbq5l\") pod \"payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt\" (UID: \"a488bb31-f27f-4d15-85a1-f11468ea3fec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt" Oct 30 12:24:48.961164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.961146 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a488bb31-f27f-4d15-85a1-f11468ea3fec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt\" (UID: \"a488bb31-f27f-4d15-85a1-f11468ea3fec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt" Oct 30 12:24:48.961576 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.961558 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661c97b4-755e-44af-b5bb-bf67720638bb-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:24:48.961705 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.961691 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661c97b4-755e-44af-b5bb-bf67720638bb-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:24:48.961814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.961801 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p8cj\" (UniqueName: \"kubernetes.io/projected/661c97b4-755e-44af-b5bb-bf67720638bb-kube-api-access-7p8cj\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:24:48.962219 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.962196 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a488bb31-f27f-4d15-85a1-f11468ea3fec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt\" (UID: \"a488bb31-f27f-4d15-85a1-f11468ea3fec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt" Oct 30 12:24:48.971936 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.971910 4413 generic.go:356] "Generic (PLEG): container finished" podID="661c97b4-755e-44af-b5bb-bf67720638bb" containerID="a39c29425bed5b3ded1fd6da679612b782ac444a855b76fbbe528b5f7296c1df" exitCode=0 Oct 30 12:24:48.972032 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.971986 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" event={"ID":"661c97b4-755e-44af-b5bb-bf67720638bb","Type":"ContainerDied","Data":"a39c29425bed5b3ded1fd6da679612b782ac444a855b76fbbe528b5f7296c1df"} Oct 30 12:24:48.972088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.972069 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" event={"ID":"661c97b4-755e-44af-b5bb-bf67720638bb","Type":"ContainerDied","Data":"2ed82d7881d96377dc87fe50f750eeead0d882bd516a40980c12df362dcd2f4a"} Oct 30 12:24:48.972133 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.972093 4413 scope.go:122] "RemoveContainer" containerID="a39c29425bed5b3ded1fd6da679612b782ac444a855b76fbbe528b5f7296c1df" Oct 30 12:24:48.972231 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.972218 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j" Oct 30 12:24:48.983268 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.983228 4413 scope.go:122] "RemoveContainer" containerID="a39c29425bed5b3ded1fd6da679612b782ac444a855b76fbbe528b5f7296c1df" Oct 30 12:24:48.983597 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:24:48.983578 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a39c29425bed5b3ded1fd6da679612b782ac444a855b76fbbe528b5f7296c1df\": container with ID starting with a39c29425bed5b3ded1fd6da679612b782ac444a855b76fbbe528b5f7296c1df not found: ID does not exist" containerID="a39c29425bed5b3ded1fd6da679612b782ac444a855b76fbbe528b5f7296c1df" Oct 30 12:24:48.983648 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.983607 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a39c29425bed5b3ded1fd6da679612b782ac444a855b76fbbe528b5f7296c1df"} err="failed to get container status \"a39c29425bed5b3ded1fd6da679612b782ac444a855b76fbbe528b5f7296c1df\": rpc error: code = NotFound desc = could not find container \"a39c29425bed5b3ded1fd6da679612b782ac444a855b76fbbe528b5f7296c1df\": container with ID starting with a39c29425bed5b3ded1fd6da679612b782ac444a855b76fbbe528b5f7296c1df not found: ID does not exist" Oct 30 12:24:48.984902 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.984883 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbq5l\" (UniqueName: \"kubernetes.io/projected/a488bb31-f27f-4d15-85a1-f11468ea3fec-kube-api-access-wbq5l\") pod \"payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt\" (UID: \"a488bb31-f27f-4d15-85a1-f11468ea3fec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt" Oct 30 12:24:48.990073 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:48.990047 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:49.041182 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:49.041109 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j"] Oct 30 12:24:49.044186 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:49.044168 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8f2898-746b477794-njl5j"] Oct 30 12:24:49.133035 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:49.132996 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:49.236226 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:49.236200 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt" Oct 30 12:24:49.415715 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:49.415680 4413 patch_prober.go:29] interesting pod/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 12:24:49.416005 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:49.415971 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" podUID="67aa5abb-dd1e-4ac9-888e-bb958fddffa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 12:24:49.527744 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:49.527709 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:49.566671 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:49.566646 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt"] Oct 30 12:24:49.567307 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:24:49.567284 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda488bb31_f27f_4d15_85a1_f11468ea3fec.slice/crio-fe13d1d3025cffd2c6c5f0e50e03e4db93336826fd50d9db74014a218c079311 WatchSource:0}: Error finding container fe13d1d3025cffd2c6c5f0e50e03e4db93336826fd50d9db74014a218c079311: Status 404 returned error can't find the container with id fe13d1d3025cffd2c6c5f0e50e03e4db93336826fd50d9db74014a218c079311 Oct 30 12:24:49.980732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:49.980695 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt" event={"ID":"a488bb31-f27f-4d15-85a1-f11468ea3fec","Type":"ContainerStarted","Data":"6f81a6b45cfac308addb37fd9dcd173aa54888f2a34ca63c7a4b313056a22cbd"} Oct 30 12:24:49.980926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:49.980752 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt" event={"ID":"a488bb31-f27f-4d15-85a1-f11468ea3fec","Type":"ContainerStarted","Data":"fe13d1d3025cffd2c6c5f0e50e03e4db93336826fd50d9db74014a218c079311"} Oct 30 12:24:49.980926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:49.980785 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt" Oct 30 12:24:50.048304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:50.048238 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt" podStartSLOduration=2.048223141 podStartE2EDuration="2.048223141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:50.044515112 +0000 UTC m=+1843.154357993" watchObservedRunningTime="2026-10-30 12:24:50.048223141 +0000 UTC m=+1843.158066022" Oct 30 12:24:50.400277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:50.394621 4413 patch_prober.go:29] interesting pod/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 12:24:50.400277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:50.394681 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" podUID="67aa5abb-dd1e-4ac9-888e-bb958fddffa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 12:24:50.869693 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:50.869602 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw"] Oct 30 12:24:50.873207 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:50.873192 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" Oct 30 12:24:50.887790 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:50.887765 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda190d0-d953-4922-a2c7-69d48381a12c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw\" (UID: \"fda190d0-d953-4922-a2c7-69d48381a12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" Oct 30 12:24:50.887936 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:50.887916 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fda190d0-d953-4922-a2c7-69d48381a12c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw\" (UID: \"fda190d0-d953-4922-a2c7-69d48381a12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" Oct 30 12:24:50.888107 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:50.888091 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm77v\" (UniqueName: \"kubernetes.io/projected/fda190d0-d953-4922-a2c7-69d48381a12c-kube-api-access-gm77v\") pod \"maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw\" (UID: \"fda190d0-d953-4922-a2c7-69d48381a12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" Oct 30 12:24:50.888283 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:50.888125 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fda190d0-d953-4922-a2c7-69d48381a12c-metrics-certs\") pod \"maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw\" (UID: \"fda190d0-d953-4922-a2c7-69d48381a12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" Oct 30 12:24:50.902495 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:50.902467 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-951ad1-serving-cert\"" Oct 30 12:24:50.926993 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:50.926957 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw"] Oct 30 12:24:50.955668 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:50.955630 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:50.989028 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:50.988999 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm77v\" (UniqueName: \"kubernetes.io/projected/fda190d0-d953-4922-a2c7-69d48381a12c-kube-api-access-gm77v\") pod \"maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw\" (UID: \"fda190d0-d953-4922-a2c7-69d48381a12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" Oct 30 12:24:50.989208 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:50.989041 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fda190d0-d953-4922-a2c7-69d48381a12c-metrics-certs\") pod \"maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw\" (UID: \"fda190d0-d953-4922-a2c7-69d48381a12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" Oct 30 12:24:50.989208 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:50.989172 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda190d0-d953-4922-a2c7-69d48381a12c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw\" (UID: \"fda190d0-d953-4922-a2c7-69d48381a12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" Oct 30 12:24:50.989374 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:50.989317 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fda190d0-d953-4922-a2c7-69d48381a12c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw\" (UID: \"fda190d0-d953-4922-a2c7-69d48381a12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" Oct 30 12:24:50.990115 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:50.990091 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda190d0-d953-4922-a2c7-69d48381a12c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw\" (UID: \"fda190d0-d953-4922-a2c7-69d48381a12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" Oct 30 12:24:51.019569 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:51.019537 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fda190d0-d953-4922-a2c7-69d48381a12c-metrics-certs\") pod \"maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw\" (UID: \"fda190d0-d953-4922-a2c7-69d48381a12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" Oct 30 12:24:51.023030 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:51.023000 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fda190d0-d953-4922-a2c7-69d48381a12c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw\" (UID: \"fda190d0-d953-4922-a2c7-69d48381a12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" Oct 30 12:24:51.024877 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:51.024812 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm77v\" (UniqueName: \"kubernetes.io/projected/fda190d0-d953-4922-a2c7-69d48381a12c-kube-api-access-gm77v\") pod \"maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw\" (UID: \"fda190d0-d953-4922-a2c7-69d48381a12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" Oct 30 12:24:51.028405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:51.028379 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="661c97b4-755e-44af-b5bb-bf67720638bb" path="/var/lib/kubelet/pods/661c97b4-755e-44af-b5bb-bf67720638bb/volumes" Oct 30 12:24:51.106550 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:51.102624 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:51.186793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:51.186762 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" Oct 30 12:24:51.416306 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:51.400615 4413 patch_prober.go:29] interesting pod/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 12:24:51.417534 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:51.417490 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" podUID="67aa5abb-dd1e-4ac9-888e-bb958fddffa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 12:24:51.533184 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:24:51.533157 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda190d0_d953_4922_a2c7_69d48381a12c.slice/crio-fced7da8b8bf58a8d990a728adf55e5dd8684ab6d7e8a41cd4f2213038cb9143 WatchSource:0}: Error finding container fced7da8b8bf58a8d990a728adf55e5dd8684ab6d7e8a41cd4f2213038cb9143: Status 404 returned error can't find the container with id fced7da8b8bf58a8d990a728adf55e5dd8684ab6d7e8a41cd4f2213038cb9143 Oct 30 12:24:51.533592 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:51.533566 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw"] Oct 30 12:24:52.003971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:52.003931 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" event={"ID":"fda190d0-d953-4922-a2c7-69d48381a12c","Type":"ContainerStarted","Data":"fced7da8b8bf58a8d990a728adf55e5dd8684ab6d7e8a41cd4f2213038cb9143"} Oct 30 12:24:52.398589 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:52.398566 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:52.398987 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:52.398954 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:52.399478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:52.399451 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:24:52.854722 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:52.854365 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:52.930699 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:52.930662 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx" Oct 30 12:24:53.025869 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:53.025837 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" event={"ID":"fda190d0-d953-4922-a2c7-69d48381a12c","Type":"ContainerStarted","Data":"bab0e867e97c07cd3f47546a3abea3474a9f42fa96f29c43167b714d1c79d9b0"} Oct 30 12:24:53.026020 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:53.025893 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" Oct 30 12:24:53.067877 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:53.067829 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" podStartSLOduration=2.670460894 podStartE2EDuration="3.06781598s" totalImagesPullingTime="397.355086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:53.065522981 +0000 UTC m=+1846.175365890" watchObservedRunningTime="2026-10-30 12:24:53.06781598 +0000 UTC m=+1846.177658862" Oct 30 12:24:55.133675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:55.133644 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:56.458503 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:56.458467 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:56.574585 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:56.574555 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:56.774022 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:56.773939 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x"] Oct 30 12:24:56.774236 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:56.774205 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" podUID="8eefa82e-f061-4c58-8fad-46c497d22c93" containerName="istio-proxy" containerID="cri-o://373d32224d2508b0fd25b1f1d8ced66b9b9288be9ccebc9e2c4fd2d40e99e54c" gracePeriod=30 Oct 30 12:24:56.804013 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:56.803983 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:56.934226 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:56.934196 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:57.080332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:57.073094 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:57.196990 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:57.196960 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:57.386191 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:57.386140 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:57.818106 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:57.818025 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:58.326002 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:58.325960 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:58.536743 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:58.536712 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:58.691447 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:58.691413 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:58.813688 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:58.813648 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:24:59.162103 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:59.162071 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" Oct 30 12:24:59.263076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:59.263046 4413 patch_prober.go:29] interesting pod/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:24:59.263319 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:59.263107 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" podUID="8eefa82e-f061-4c58-8fad-46c497d22c93" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:24:59.633468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:24:59.633429 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:00.510738 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:00.510706 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:00.637988 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:00.637953 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:00.999662 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:00.999634 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt" Oct 30 12:25:01.739785 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:01.739748 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:02.076165 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.076136 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:25:02.168075 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.168039 4413 generic.go:356] "Generic (PLEG): container finished" podID="8eefa82e-f061-4c58-8fad-46c497d22c93" containerID="373d32224d2508b0fd25b1f1d8ced66b9b9288be9ccebc9e2c4fd2d40e99e54c" exitCode=0 Oct 30 12:25:02.168288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.168089 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" event={"ID":"8eefa82e-f061-4c58-8fad-46c497d22c93","Type":"ContainerDied","Data":"373d32224d2508b0fd25b1f1d8ced66b9b9288be9ccebc9e2c4fd2d40e99e54c"} Oct 30 12:25:02.168288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.168107 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" Oct 30 12:25:02.168288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.168133 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x" event={"ID":"8eefa82e-f061-4c58-8fad-46c497d22c93","Type":"ContainerDied","Data":"9b34d77720f15fede23ee27086f29512161cc7a7c57296be5ff039421927cfb2"} Oct 30 12:25:02.168288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.168161 4413 scope.go:122] "RemoveContainer" containerID="373d32224d2508b0fd25b1f1d8ced66b9b9288be9ccebc9e2c4fd2d40e99e54c" Oct 30 12:25:02.174724 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.174699 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-podinfo\") pod \"8eefa82e-f061-4c58-8fad-46c497d22c93\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " Oct 30 12:25:02.174897 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.174731 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eefa82e-f061-4c58-8fad-46c497d22c93-kube-api-access-xlxlq\" (UniqueName: \"kubernetes.io/projected/8eefa82e-f061-4c58-8fad-46c497d22c93-kube-api-access-xlxlq\") pod \"8eefa82e-f061-4c58-8fad-46c497d22c93\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " Oct 30 12:25:02.174897 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.174844 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-token\" (UniqueName: \"kubernetes.io/projected/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-token\") pod \"8eefa82e-f061-4c58-8fad-46c497d22c93\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " Oct 30 12:25:02.175022 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.174899 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eefa82e-f061-4c58-8fad-46c497d22c93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eefa82e-f061-4c58-8fad-46c497d22c93-istiod-ca-cert\") pod \"8eefa82e-f061-4c58-8fad-46c497d22c93\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " Oct 30 12:25:02.175022 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.174945 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-workload-socket\") pod \"8eefa82e-f061-4c58-8fad-46c497d22c93\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " Oct 30 12:25:02.175022 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.174995 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-workload-certs\") pod \"8eefa82e-f061-4c58-8fad-46c497d22c93\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " Oct 30 12:25:02.175180 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.175030 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-envoy\") pod \"8eefa82e-f061-4c58-8fad-46c497d22c93\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " Oct 30 12:25:02.175180 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.175074 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-credential-socket\") pod \"8eefa82e-f061-4c58-8fad-46c497d22c93\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " Oct 30 12:25:02.175180 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.175098 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-data\") pod \"8eefa82e-f061-4c58-8fad-46c497d22c93\" (UID: \"8eefa82e-f061-4c58-8fad-46c497d22c93\") " Oct 30 12:25:02.177658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.175852 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-workload-socket" pod "8eefa82e-f061-4c58-8fad-46c497d22c93" (UID: "8eefa82e-f061-4c58-8fad-46c497d22c93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:02.177658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.176061 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-data" pod "8eefa82e-f061-4c58-8fad-46c497d22c93" (UID: "8eefa82e-f061-4c58-8fad-46c497d22c93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:02.177658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.176539 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:02.177658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.176562 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:02.177658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.177499 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eefa82e-f061-4c58-8fad-46c497d22c93-istiod-ca-cert" pod "8eefa82e-f061-4c58-8fad-46c497d22c93" (UID: "8eefa82e-f061-4c58-8fad-46c497d22c93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:02.178365 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.178346 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-credential-socket" pod "8eefa82e-f061-4c58-8fad-46c497d22c93" (UID: "8eefa82e-f061-4c58-8fad-46c497d22c93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:02.178846 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.178823 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eefa82e-f061-4c58-8fad-46c497d22c93-kube-api-access-xlxlq" pod "8eefa82e-f061-4c58-8fad-46c497d22c93" (UID: "8eefa82e-f061-4c58-8fad-46c497d22c93"). InnerVolumeSpecName "kube-api-access-xlxlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:02.178971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.178951 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-workload-certs" pod "8eefa82e-f061-4c58-8fad-46c497d22c93" (UID: "8eefa82e-f061-4c58-8fad-46c497d22c93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:02.181015 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.180895 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-podinfo" pod "8eefa82e-f061-4c58-8fad-46c497d22c93" (UID: "8eefa82e-f061-4c58-8fad-46c497d22c93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:25:02.181119 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.181097 4413 scope.go:122] "RemoveContainer" containerID="0a072c9ced35c544026c69ff8b64517f747cdfc8dd7728ee62797891110a61d4" Oct 30 12:25:02.182996 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.182972 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-token" pod "8eefa82e-f061-4c58-8fad-46c497d22c93" (UID: "8eefa82e-f061-4c58-8fad-46c497d22c93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:02.183468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.183445 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-envoy" pod "8eefa82e-f061-4c58-8fad-46c497d22c93" (UID: "8eefa82e-f061-4c58-8fad-46c497d22c93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:02.214314 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.214284 4413 scope.go:122] "RemoveContainer" containerID="373d32224d2508b0fd25b1f1d8ced66b9b9288be9ccebc9e2c4fd2d40e99e54c" Oct 30 12:25:02.214678 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:25:02.214650 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373d32224d2508b0fd25b1f1d8ced66b9b9288be9ccebc9e2c4fd2d40e99e54c\": container with ID starting with 373d32224d2508b0fd25b1f1d8ced66b9b9288be9ccebc9e2c4fd2d40e99e54c not found: ID does not exist" containerID="373d32224d2508b0fd25b1f1d8ced66b9b9288be9ccebc9e2c4fd2d40e99e54c" Oct 30 12:25:02.214745 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.214691 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373d32224d2508b0fd25b1f1d8ced66b9b9288be9ccebc9e2c4fd2d40e99e54c"} err="failed to get container status \"373d32224d2508b0fd25b1f1d8ced66b9b9288be9ccebc9e2c4fd2d40e99e54c\": rpc error: code = NotFound desc = could not find container \"373d32224d2508b0fd25b1f1d8ced66b9b9288be9ccebc9e2c4fd2d40e99e54c\": container with ID starting with 373d32224d2508b0fd25b1f1d8ced66b9b9288be9ccebc9e2c4fd2d40e99e54c not found: ID does not exist" Oct 30 12:25:02.214745 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.214710 4413 scope.go:122] "RemoveContainer" containerID="0a072c9ced35c544026c69ff8b64517f747cdfc8dd7728ee62797891110a61d4" Oct 30 12:25:02.215016 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:25:02.215001 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a072c9ced35c544026c69ff8b64517f747cdfc8dd7728ee62797891110a61d4\": container with ID starting with 0a072c9ced35c544026c69ff8b64517f747cdfc8dd7728ee62797891110a61d4 not found: ID does not exist" containerID="0a072c9ced35c544026c69ff8b64517f747cdfc8dd7728ee62797891110a61d4" Oct 30 12:25:02.215063 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.215019 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a072c9ced35c544026c69ff8b64517f747cdfc8dd7728ee62797891110a61d4"} err="failed to get container status \"0a072c9ced35c544026c69ff8b64517f747cdfc8dd7728ee62797891110a61d4\": rpc error: code = NotFound desc = could not find container \"0a072c9ced35c544026c69ff8b64517f747cdfc8dd7728ee62797891110a61d4\": container with ID starting with 0a072c9ced35c544026c69ff8b64517f747cdfc8dd7728ee62797891110a61d4 not found: ID does not exist" Oct 30 12:25:02.277833 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.277800 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eefa82e-f061-4c58-8fad-46c497d22c93-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:02.277833 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.277835 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:02.278038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.277845 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:02.278038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.277853 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eefa82e-f061-4c58-8fad-46c497d22c93-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:02.278038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.277862 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:02.278038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.277870 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlxlq\" (UniqueName: \"kubernetes.io/projected/8eefa82e-f061-4c58-8fad-46c497d22c93-kube-api-access-xlxlq\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:02.278038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.277878 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eefa82e-f061-4c58-8fad-46c497d22c93-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:02.514892 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.514858 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x"] Oct 30 12:25:02.527064 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:02.527034 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8f2898-openshift-default-64d4dc6f85-gg25x"] Oct 30 12:25:03.021516 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:03.021482 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eefa82e-f061-4c58-8fad-46c497d22c93" path="/var/lib/kubelet/pods/8eefa82e-f061-4c58-8fad-46c497d22c93/volumes" Oct 30 12:25:03.087610 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:03.087558 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:03.211180 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:03.211148 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:03.811630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:03.811596 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:03.950169 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:03.950125 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:04.763674 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:04.763625 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:04.887092 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:04.887055 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:05.817958 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:25:05.817921 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:25:06.059496 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:06.059456 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:06.441774 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:06.441739 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:07.754283 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:07.754175 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:09.346698 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:09.346665 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:11.084675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:11.084636 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:11.845340 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:11.845305 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft"] Oct 30 12:25:11.845571 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:11.845536 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" podUID="7995a1f6-7cff-4b6a-b612-78b31bad73ba" containerName="istio-proxy" containerID="cri-o://3a8459d7adeca702e5713789eaa1ed453a6e8cdf7bb8dc58051f3848769399a8" gracePeriod=30 Oct 30 12:25:11.856945 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:11.856911 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m"] Oct 30 12:25:11.857155 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:11.857132 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" podUID="1e53a452-5a36-4d78-9a14-35f6ebb715ba" containerName="istio-proxy" containerID="cri-o://880dba97d5d51da4be6f2467770faa1db4d132997610a073ca1367aacfbf77a6" gracePeriod=30 Oct 30 12:25:11.860869 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:11.860842 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk"] Oct 30 12:25:11.861083 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:11.861043 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" podUID="f7b5b44c-e86c-4990-a68b-1dbff2e42a46" containerName="istio-proxy" containerID="cri-o://abdd6770f5991b4437c69c662f965343b58521cf0740ecc18cb7a9a3072ce74a" gracePeriod=30 Oct 30 12:25:11.868354 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:11.868333 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj"] Oct 30 12:25:11.868521 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:11.868503 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" podUID="3a8cf876-a088-4dd2-9890-27772782da71" containerName="istio-proxy" containerID="cri-o://5bcd85c8fec1de4937e11c880f4f2d41a12d73824ec12fe69d89869c1dc1f593" gracePeriod=30 Oct 30 12:25:12.789330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:12.789290 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:13.846631 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:13.846592 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx"] Oct 30 12:25:14.016802 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:14.016469 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:14.148849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:14.148302 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:14.276922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:14.276891 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:14.553454 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:14.553374 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:14.812011 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:14.811918 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:14.928924 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:14.928891 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:16.434535 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:16.434409 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:16.547325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:16.547139 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:16.714484 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:16.714358 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:17.211927 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.211902 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:25:17.214038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.214016 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:25:17.290463 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.290437 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:25:17.301723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.301696 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:25:17.387424 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.387392 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-workload-socket\") pod \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " Oct 30 12:25:17.387595 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.387459 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-workload-socket\") pod \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " Oct 30 12:25:17.387595 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.387505 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-workload-certs\") pod \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " Oct 30 12:25:17.387595 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.387544 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istiod-ca-cert\") pod \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " Oct 30 12:25:17.387595 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.387578 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-credential-socket\") pod \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " Oct 30 12:25:17.387748 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.387618 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-podinfo\") pod \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " Oct 30 12:25:17.387748 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.387648 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-envoy\") pod \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " Oct 30 12:25:17.387748 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.387685 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-envoy\") pod \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " Oct 30 12:25:17.387855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.387751 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-data\") pod \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " Oct 30 12:25:17.387855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.387754 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-workload-socket" pod "7995a1f6-7cff-4b6a-b612-78b31bad73ba" (UID: "7995a1f6-7cff-4b6a-b612-78b31bad73ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.387855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.387800 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-workload-socket\") pod \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " Oct 30 12:25:17.387855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.387833 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-token\" (UniqueName: \"kubernetes.io/projected/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-token\") pod \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " Oct 30 12:25:17.387991 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.387868 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-workload-certs\") pod \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " Oct 30 12:25:17.387991 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.387918 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-envoy\") pod \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " Oct 30 12:25:17.387991 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.387944 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-workload-certs\") pod \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " Oct 30 12:25:17.387991 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.387982 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istiod-ca-cert\") pod \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " Oct 30 12:25:17.388149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.388002 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-credential-socket" pod "f7b5b44c-e86c-4990-a68b-1dbff2e42a46" (UID: "f7b5b44c-e86c-4990-a68b-1dbff2e42a46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.388149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.388009 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-data\") pod \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " Oct 30 12:25:17.388149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.388083 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istiod-ca-cert\") pod \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " Oct 30 12:25:17.388316 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.388154 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7995a1f6-7cff-4b6a-b612-78b31bad73ba-kube-api-access-cwbt8\" (UniqueName: \"kubernetes.io/projected/7995a1f6-7cff-4b6a-b612-78b31bad73ba-kube-api-access-cwbt8\") pod \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " Oct 30 12:25:17.388316 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.388198 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-credential-socket\") pod \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " Oct 30 12:25:17.388316 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.388231 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-podinfo\") pod \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " Oct 30 12:25:17.388316 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.388310 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-podinfo\") pod \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " Oct 30 12:25:17.388536 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.388352 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-credential-socket\") pod \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " Oct 30 12:25:17.388536 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.388466 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-kube-api-access-khgzc\" (UniqueName: \"kubernetes.io/projected/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-kube-api-access-khgzc\") pod \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " Oct 30 12:25:17.388536 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.388502 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-token\" (UniqueName: \"kubernetes.io/projected/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-token\") pod \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\" (UID: \"f7b5b44c-e86c-4990-a68b-1dbff2e42a46\") " Oct 30 12:25:17.388692 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.388550 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-token\" (UniqueName: \"kubernetes.io/projected/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-token\") pod \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " Oct 30 12:25:17.388692 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.388618 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-data\") pod \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\" (UID: \"7995a1f6-7cff-4b6a-b612-78b31bad73ba\") " Oct 30 12:25:17.388806 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.388718 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-data" pod "f7b5b44c-e86c-4990-a68b-1dbff2e42a46" (UID: "f7b5b44c-e86c-4990-a68b-1dbff2e42a46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.388806 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.388748 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e53a452-5a36-4d78-9a14-35f6ebb715ba-kube-api-access-mpqrz\" (UniqueName: \"kubernetes.io/projected/1e53a452-5a36-4d78-9a14-35f6ebb715ba-kube-api-access-mpqrz\") pod \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\" (UID: \"1e53a452-5a36-4d78-9a14-35f6ebb715ba\") " Oct 30 12:25:17.389862 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.389834 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.389991 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.389865 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.389991 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.389881 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.391591 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.391563 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-podinfo" pod "7995a1f6-7cff-4b6a-b612-78b31bad73ba" (UID: "7995a1f6-7cff-4b6a-b612-78b31bad73ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:25:17.391798 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.391779 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-workload-certs" pod "1e53a452-5a36-4d78-9a14-35f6ebb715ba" (UID: "1e53a452-5a36-4d78-9a14-35f6ebb715ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.394930 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.394858 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-token" pod "1e53a452-5a36-4d78-9a14-35f6ebb715ba" (UID: "1e53a452-5a36-4d78-9a14-35f6ebb715ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:17.395633 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.395611 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istiod-ca-cert" pod "f7b5b44c-e86c-4990-a68b-1dbff2e42a46" (UID: "f7b5b44c-e86c-4990-a68b-1dbff2e42a46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:17.396054 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.395970 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-credential-socket" pod "1e53a452-5a36-4d78-9a14-35f6ebb715ba" (UID: "1e53a452-5a36-4d78-9a14-35f6ebb715ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.396054 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.396022 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istiod-ca-cert" pod "1e53a452-5a36-4d78-9a14-35f6ebb715ba" (UID: "1e53a452-5a36-4d78-9a14-35f6ebb715ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:17.396216 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.396084 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-workload-certs" pod "7995a1f6-7cff-4b6a-b612-78b31bad73ba" (UID: "7995a1f6-7cff-4b6a-b612-78b31bad73ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.396216 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.396191 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istiod-ca-cert" pod "7995a1f6-7cff-4b6a-b612-78b31bad73ba" (UID: "7995a1f6-7cff-4b6a-b612-78b31bad73ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:17.396358 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.396343 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-data" pod "7995a1f6-7cff-4b6a-b612-78b31bad73ba" (UID: "7995a1f6-7cff-4b6a-b612-78b31bad73ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.396620 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.396519 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-data" pod "1e53a452-5a36-4d78-9a14-35f6ebb715ba" (UID: "1e53a452-5a36-4d78-9a14-35f6ebb715ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.396874 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.396713 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-workload-socket" pod "1e53a452-5a36-4d78-9a14-35f6ebb715ba" (UID: "1e53a452-5a36-4d78-9a14-35f6ebb715ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.399037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.396873 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-credential-socket" pod "7995a1f6-7cff-4b6a-b612-78b31bad73ba" (UID: "7995a1f6-7cff-4b6a-b612-78b31bad73ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.399037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.397120 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-envoy" pod "1e53a452-5a36-4d78-9a14-35f6ebb715ba" (UID: "1e53a452-5a36-4d78-9a14-35f6ebb715ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.399037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.398639 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-workload-socket" pod "f7b5b44c-e86c-4990-a68b-1dbff2e42a46" (UID: "f7b5b44c-e86c-4990-a68b-1dbff2e42a46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.399948 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.399521 4413 generic.go:356] "Generic (PLEG): container finished" podID="3a8cf876-a088-4dd2-9890-27772782da71" containerID="5bcd85c8fec1de4937e11c880f4f2d41a12d73824ec12fe69d89869c1dc1f593" exitCode=0 Oct 30 12:25:17.399948 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.399559 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" event={"ID":"3a8cf876-a088-4dd2-9890-27772782da71","Type":"ContainerDied","Data":"5bcd85c8fec1de4937e11c880f4f2d41a12d73824ec12fe69d89869c1dc1f593"} Oct 30 12:25:17.399948 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.399631 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" event={"ID":"3a8cf876-a088-4dd2-9890-27772782da71","Type":"ContainerDied","Data":"3bcaa8576049114c4ae96d990119e28cd8eb2af2ae498cc5d72befcda22d333f"} Oct 30 12:25:17.399948 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.399766 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj" Oct 30 12:25:17.399948 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.399802 4413 scope.go:122] "RemoveContainer" containerID="5bcd85c8fec1de4937e11c880f4f2d41a12d73824ec12fe69d89869c1dc1f593" Oct 30 12:25:17.400302 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.400128 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-workload-certs" pod "f7b5b44c-e86c-4990-a68b-1dbff2e42a46" (UID: "f7b5b44c-e86c-4990-a68b-1dbff2e42a46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.401527 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.401111 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-envoy" pod "7995a1f6-7cff-4b6a-b612-78b31bad73ba" (UID: "7995a1f6-7cff-4b6a-b612-78b31bad73ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.401733 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.401707 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-podinfo" pod "f7b5b44c-e86c-4990-a68b-1dbff2e42a46" (UID: "f7b5b44c-e86c-4990-a68b-1dbff2e42a46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:25:17.401806 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.401735 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-token" pod "7995a1f6-7cff-4b6a-b612-78b31bad73ba" (UID: "7995a1f6-7cff-4b6a-b612-78b31bad73ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:17.401806 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.401774 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-kube-api-access-khgzc" pod "f7b5b44c-e86c-4990-a68b-1dbff2e42a46" (UID: "f7b5b44c-e86c-4990-a68b-1dbff2e42a46"). InnerVolumeSpecName "kube-api-access-khgzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:17.401806 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.401795 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e53a452-5a36-4d78-9a14-35f6ebb715ba-kube-api-access-mpqrz" pod "1e53a452-5a36-4d78-9a14-35f6ebb715ba" (UID: "1e53a452-5a36-4d78-9a14-35f6ebb715ba"). InnerVolumeSpecName "kube-api-access-mpqrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:17.401979 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.401843 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-podinfo" pod "1e53a452-5a36-4d78-9a14-35f6ebb715ba" (UID: "1e53a452-5a36-4d78-9a14-35f6ebb715ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:25:17.404368 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.402381 4413 generic.go:356] "Generic (PLEG): container finished" podID="f7b5b44c-e86c-4990-a68b-1dbff2e42a46" containerID="abdd6770f5991b4437c69c662f965343b58521cf0740ecc18cb7a9a3072ce74a" exitCode=0 Oct 30 12:25:17.404368 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.402485 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" event={"ID":"f7b5b44c-e86c-4990-a68b-1dbff2e42a46","Type":"ContainerDied","Data":"abdd6770f5991b4437c69c662f965343b58521cf0740ecc18cb7a9a3072ce74a"} Oct 30 12:25:17.404368 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.402498 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" Oct 30 12:25:17.404368 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.402527 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk" event={"ID":"f7b5b44c-e86c-4990-a68b-1dbff2e42a46","Type":"ContainerDied","Data":"95b3a0658dfc10061b9214061a4712ffac3256ff944602d70c550e2b0661c6e6"} Oct 30 12:25:17.404368 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.402528 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-token" pod "f7b5b44c-e86c-4990-a68b-1dbff2e42a46" (UID: "f7b5b44c-e86c-4990-a68b-1dbff2e42a46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:17.404643 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.404558 4413 generic.go:356] "Generic (PLEG): container finished" podID="7995a1f6-7cff-4b6a-b612-78b31bad73ba" containerID="3a8459d7adeca702e5713789eaa1ed453a6e8cdf7bb8dc58051f3848769399a8" exitCode=0 Oct 30 12:25:17.404691 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.404679 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" Oct 30 12:25:17.405094 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.404824 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" event={"ID":"7995a1f6-7cff-4b6a-b612-78b31bad73ba","Type":"ContainerDied","Data":"3a8459d7adeca702e5713789eaa1ed453a6e8cdf7bb8dc58051f3848769399a8"} Oct 30 12:25:17.405094 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.404873 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft" event={"ID":"7995a1f6-7cff-4b6a-b612-78b31bad73ba","Type":"ContainerDied","Data":"4e3340392dc7cb684fb8fbd4cb0120c33e27772ae2590f719e0c6234b015e095"} Oct 30 12:25:17.417732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.406411 4413 generic.go:356] "Generic (PLEG): container finished" podID="1e53a452-5a36-4d78-9a14-35f6ebb715ba" containerID="880dba97d5d51da4be6f2467770faa1db4d132997610a073ca1367aacfbf77a6" exitCode=0 Oct 30 12:25:17.417732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.406494 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" Oct 30 12:25:17.417732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.406494 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" event={"ID":"1e53a452-5a36-4d78-9a14-35f6ebb715ba","Type":"ContainerDied","Data":"880dba97d5d51da4be6f2467770faa1db4d132997610a073ca1367aacfbf77a6"} Oct 30 12:25:17.417732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.406615 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m" event={"ID":"1e53a452-5a36-4d78-9a14-35f6ebb715ba","Type":"ContainerDied","Data":"b95f8ecf9c269311abefcd0529eacc79c756094f5f514a6e8eeaffe04ac04653"} Oct 30 12:25:17.417732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.406936 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-envoy" pod "f7b5b44c-e86c-4990-a68b-1dbff2e42a46" (UID: "f7b5b44c-e86c-4990-a68b-1dbff2e42a46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.417732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.407120 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7995a1f6-7cff-4b6a-b612-78b31bad73ba-kube-api-access-cwbt8" pod "7995a1f6-7cff-4b6a-b612-78b31bad73ba" (UID: "7995a1f6-7cff-4b6a-b612-78b31bad73ba"). InnerVolumeSpecName "kube-api-access-cwbt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:17.417732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.415521 4413 scope.go:122] "RemoveContainer" containerID="5bcd85c8fec1de4937e11c880f4f2d41a12d73824ec12fe69d89869c1dc1f593" Oct 30 12:25:17.417732 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:25:17.415900 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bcd85c8fec1de4937e11c880f4f2d41a12d73824ec12fe69d89869c1dc1f593\": container with ID starting with 5bcd85c8fec1de4937e11c880f4f2d41a12d73824ec12fe69d89869c1dc1f593 not found: ID does not exist" containerID="5bcd85c8fec1de4937e11c880f4f2d41a12d73824ec12fe69d89869c1dc1f593" Oct 30 12:25:17.417732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.415935 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bcd85c8fec1de4937e11c880f4f2d41a12d73824ec12fe69d89869c1dc1f593"} err="failed to get container status \"5bcd85c8fec1de4937e11c880f4f2d41a12d73824ec12fe69d89869c1dc1f593\": rpc error: code = NotFound desc = could not find container \"5bcd85c8fec1de4937e11c880f4f2d41a12d73824ec12fe69d89869c1dc1f593\": container with ID starting with 5bcd85c8fec1de4937e11c880f4f2d41a12d73824ec12fe69d89869c1dc1f593 not found: ID does not exist" Oct 30 12:25:17.417732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.415958 4413 scope.go:122] "RemoveContainer" containerID="abdd6770f5991b4437c69c662f965343b58521cf0740ecc18cb7a9a3072ce74a" Oct 30 12:25:17.441162 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.441142 4413 scope.go:122] "RemoveContainer" containerID="abdd6770f5991b4437c69c662f965343b58521cf0740ecc18cb7a9a3072ce74a" Oct 30 12:25:17.441595 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:25:17.441570 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abdd6770f5991b4437c69c662f965343b58521cf0740ecc18cb7a9a3072ce74a\": container with ID starting with abdd6770f5991b4437c69c662f965343b58521cf0740ecc18cb7a9a3072ce74a not found: ID does not exist" containerID="abdd6770f5991b4437c69c662f965343b58521cf0740ecc18cb7a9a3072ce74a" Oct 30 12:25:17.441667 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.441608 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abdd6770f5991b4437c69c662f965343b58521cf0740ecc18cb7a9a3072ce74a"} err="failed to get container status \"abdd6770f5991b4437c69c662f965343b58521cf0740ecc18cb7a9a3072ce74a\": rpc error: code = NotFound desc = could not find container \"abdd6770f5991b4437c69c662f965343b58521cf0740ecc18cb7a9a3072ce74a\": container with ID starting with abdd6770f5991b4437c69c662f965343b58521cf0740ecc18cb7a9a3072ce74a not found: ID does not exist" Oct 30 12:25:17.441667 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.441632 4413 scope.go:122] "RemoveContainer" containerID="3a8459d7adeca702e5713789eaa1ed453a6e8cdf7bb8dc58051f3848769399a8" Oct 30 12:25:17.457135 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.457112 4413 scope.go:122] "RemoveContainer" containerID="3a8459d7adeca702e5713789eaa1ed453a6e8cdf7bb8dc58051f3848769399a8" Oct 30 12:25:17.457640 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:25:17.457502 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8459d7adeca702e5713789eaa1ed453a6e8cdf7bb8dc58051f3848769399a8\": container with ID starting with 3a8459d7adeca702e5713789eaa1ed453a6e8cdf7bb8dc58051f3848769399a8 not found: ID does not exist" containerID="3a8459d7adeca702e5713789eaa1ed453a6e8cdf7bb8dc58051f3848769399a8" Oct 30 12:25:17.457640 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.457536 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8459d7adeca702e5713789eaa1ed453a6e8cdf7bb8dc58051f3848769399a8"} err="failed to get container status \"3a8459d7adeca702e5713789eaa1ed453a6e8cdf7bb8dc58051f3848769399a8\": rpc error: code = NotFound desc = could not find container \"3a8459d7adeca702e5713789eaa1ed453a6e8cdf7bb8dc58051f3848769399a8\": container with ID starting with 3a8459d7adeca702e5713789eaa1ed453a6e8cdf7bb8dc58051f3848769399a8 not found: ID does not exist" Oct 30 12:25:17.457640 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.457560 4413 scope.go:122] "RemoveContainer" containerID="880dba97d5d51da4be6f2467770faa1db4d132997610a073ca1367aacfbf77a6" Oct 30 12:25:17.472746 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.472712 4413 scope.go:122] "RemoveContainer" containerID="880dba97d5d51da4be6f2467770faa1db4d132997610a073ca1367aacfbf77a6" Oct 30 12:25:17.473131 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:25:17.473104 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880dba97d5d51da4be6f2467770faa1db4d132997610a073ca1367aacfbf77a6\": container with ID starting with 880dba97d5d51da4be6f2467770faa1db4d132997610a073ca1367aacfbf77a6 not found: ID does not exist" containerID="880dba97d5d51da4be6f2467770faa1db4d132997610a073ca1367aacfbf77a6" Oct 30 12:25:17.473231 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.473142 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880dba97d5d51da4be6f2467770faa1db4d132997610a073ca1367aacfbf77a6"} err="failed to get container status \"880dba97d5d51da4be6f2467770faa1db4d132997610a073ca1367aacfbf77a6\": rpc error: code = NotFound desc = could not find container \"880dba97d5d51da4be6f2467770faa1db4d132997610a073ca1367aacfbf77a6\": container with ID starting with 880dba97d5d51da4be6f2467770faa1db4d132997610a073ca1367aacfbf77a6 not found: ID does not exist" Oct 30 12:25:17.490425 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.490398 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-istio-data\") pod \"3a8cf876-a088-4dd2-9890-27772782da71\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " Oct 30 12:25:17.490565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.490469 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a8cf876-a088-4dd2-9890-27772782da71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a8cf876-a088-4dd2-9890-27772782da71-istio-podinfo\") pod \"3a8cf876-a088-4dd2-9890-27772782da71\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " Oct 30 12:25:17.490657 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.490641 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-istio-envoy\") pod \"3a8cf876-a088-4dd2-9890-27772782da71\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " Oct 30 12:25:17.490754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.490728 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-istio-data" pod "3a8cf876-a088-4dd2-9890-27772782da71" (UID: "3a8cf876-a088-4dd2-9890-27772782da71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.493663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.493636 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a8cf876-a088-4dd2-9890-27772782da71-kube-api-access-78cwb\" (UniqueName: \"kubernetes.io/projected/3a8cf876-a088-4dd2-9890-27772782da71-kube-api-access-78cwb\") pod \"3a8cf876-a088-4dd2-9890-27772782da71\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " Oct 30 12:25:17.493800 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.493695 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a8cf876-a088-4dd2-9890-27772782da71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a8cf876-a088-4dd2-9890-27772782da71-istiod-ca-cert\") pod \"3a8cf876-a088-4dd2-9890-27772782da71\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " Oct 30 12:25:17.493800 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.493737 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-workload-certs\") pod \"3a8cf876-a088-4dd2-9890-27772782da71\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " Oct 30 12:25:17.493800 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.493795 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-workload-socket\") pod \"3a8cf876-a088-4dd2-9890-27772782da71\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " Oct 30 12:25:17.493967 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.493901 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-credential-socket\") pod \"3a8cf876-a088-4dd2-9890-27772782da71\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " Oct 30 12:25:17.493967 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.493938 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a8cf876-a088-4dd2-9890-27772782da71-istio-token\" (UniqueName: \"kubernetes.io/projected/3a8cf876-a088-4dd2-9890-27772782da71-istio-token\") pod \"3a8cf876-a088-4dd2-9890-27772782da71\" (UID: \"3a8cf876-a088-4dd2-9890-27772782da71\") " Oct 30 12:25:17.494575 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.494305 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-workload-certs" pod "3a8cf876-a088-4dd2-9890-27772782da71" (UID: "3a8cf876-a088-4dd2-9890-27772782da71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.494654 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.494617 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-credential-socket" pod "3a8cf876-a088-4dd2-9890-27772782da71" (UID: "3a8cf876-a088-4dd2-9890-27772782da71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.494931 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.494898 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a8cf876-a088-4dd2-9890-27772782da71-istiod-ca-cert" pod "3a8cf876-a088-4dd2-9890-27772782da71" (UID: "3a8cf876-a088-4dd2-9890-27772782da71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:17.495402 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.495127 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.495574 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.495560 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.495660 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.495650 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.496351 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.496337 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.497711 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497696 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwbt8\" (UniqueName: \"kubernetes.io/projected/7995a1f6-7cff-4b6a-b612-78b31bad73ba-kube-api-access-cwbt8\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.497711 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.495784 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8cf876-a088-4dd2-9890-27772782da71-kube-api-access-78cwb" pod "3a8cf876-a088-4dd2-9890-27772782da71" (UID: "3a8cf876-a088-4dd2-9890-27772782da71"). InnerVolumeSpecName "kube-api-access-78cwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:17.497952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.496886 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-workload-socket" pod "3a8cf876-a088-4dd2-9890-27772782da71" (UID: "3a8cf876-a088-4dd2-9890-27772782da71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.497952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497515 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8cf876-a088-4dd2-9890-27772782da71-istio-token" pod "3a8cf876-a088-4dd2-9890-27772782da71" (UID: "3a8cf876-a088-4dd2-9890-27772782da71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:17.497952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497597 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-istio-envoy" pod "3a8cf876-a088-4dd2-9890-27772782da71" (UID: "3a8cf876-a088-4dd2-9890-27772782da71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:17.497952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497642 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a8cf876-a088-4dd2-9890-27772782da71-istio-podinfo" pod "3a8cf876-a088-4dd2-9890-27772782da71" (UID: "3a8cf876-a088-4dd2-9890-27772782da71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:25:17.497952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497774 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.497952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497792 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.497952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497805 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.497952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497819 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.497952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497832 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khgzc\" (UniqueName: \"kubernetes.io/projected/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-kube-api-access-khgzc\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.497952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497845 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.497952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497858 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.497952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497869 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.497952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497883 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a8cf876-a088-4dd2-9890-27772782da71-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.497952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497896 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpqrz\" (UniqueName: \"kubernetes.io/projected/1e53a452-5a36-4d78-9a14-35f6ebb715ba-kube-api-access-mpqrz\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.497952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497909 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.497952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497922 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.497952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497937 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.497952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497950 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.498716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497963 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.498716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497975 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.498716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497987 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7995a1f6-7cff-4b6a-b612-78b31bad73ba-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.498716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.497999 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b5b44c-e86c-4990-a68b-1dbff2e42a46-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.498716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.498012 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.498716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.498025 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.498716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.498037 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.498716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.498048 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e53a452-5a36-4d78-9a14-35f6ebb715ba-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.498716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.498062 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e53a452-5a36-4d78-9a14-35f6ebb715ba-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.499309 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.499285 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m"] Oct 30 12:25:17.502750 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.502731 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-vs96m"] Oct 30 12:25:17.599144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.599047 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.599144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.599082 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78cwb\" (UniqueName: \"kubernetes.io/projected/3a8cf876-a088-4dd2-9890-27772782da71-kube-api-access-78cwb\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.599144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.599092 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf876-a088-4dd2-9890-27772782da71-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.599144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.599101 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a8cf876-a088-4dd2-9890-27772782da71-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.599144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.599110 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a8cf876-a088-4dd2-9890-27772782da71-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:17.761375 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.761302 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj"] Oct 30 12:25:17.768116 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.768072 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-2mzqj"] Oct 30 12:25:17.813285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.813212 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft"] Oct 30 12:25:17.825907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.825829 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-lxtft"] Oct 30 12:25:17.847059 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.847025 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk"] Oct 30 12:25:17.851221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:17.851141 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f920f7-openshift-default-7c9767c97-svfjk"] Oct 30 12:25:18.350837 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:18.350797 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:18.847465 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:18.847417 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx" podUID="dfcc37aa-99af-4a3c-bf39-34d2118506e6" containerName="payload-processing" containerID="cri-o://2b187a83eb5fb68a5cd7c5fe99b9b160d25a39d1a74393995f52fcbc04a24fa6" gracePeriod=25 Oct 30 12:25:19.024314 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.023499 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e53a452-5a36-4d78-9a14-35f6ebb715ba" path="/var/lib/kubelet/pods/1e53a452-5a36-4d78-9a14-35f6ebb715ba/volumes" Oct 30 12:25:19.024314 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.024224 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a8cf876-a088-4dd2-9890-27772782da71" path="/var/lib/kubelet/pods/3a8cf876-a088-4dd2-9890-27772782da71/volumes" Oct 30 12:25:19.024825 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.024804 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7995a1f6-7cff-4b6a-b612-78b31bad73ba" path="/var/lib/kubelet/pods/7995a1f6-7cff-4b6a-b612-78b31bad73ba/volumes" Oct 30 12:25:19.025803 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.025780 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b5b44c-e86c-4990-a68b-1dbff2e42a46" path="/var/lib/kubelet/pods/f7b5b44c-e86c-4990-a68b-1dbff2e42a46/volumes" Oct 30 12:25:19.150706 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.150679 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx" Oct 30 12:25:19.232694 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.232666 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfcc37aa-99af-4a3c-bf39-34d2118506e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcc37aa-99af-4a3c-bf39-34d2118506e6-plugins-config-volume\") pod \"dfcc37aa-99af-4a3c-bf39-34d2118506e6\" (UID: \"dfcc37aa-99af-4a3c-bf39-34d2118506e6\") " Oct 30 12:25:19.232950 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.232737 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfcc37aa-99af-4a3c-bf39-34d2118506e6-kube-api-access-4jchd\" (UniqueName: \"kubernetes.io/projected/dfcc37aa-99af-4a3c-bf39-34d2118506e6-kube-api-access-4jchd\") pod \"dfcc37aa-99af-4a3c-bf39-34d2118506e6\" (UID: \"dfcc37aa-99af-4a3c-bf39-34d2118506e6\") " Oct 30 12:25:19.233108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.233084 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfcc37aa-99af-4a3c-bf39-34d2118506e6-plugins-config-volume" pod "dfcc37aa-99af-4a3c-bf39-34d2118506e6" (UID: "dfcc37aa-99af-4a3c-bf39-34d2118506e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:19.233480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.233458 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcc37aa-99af-4a3c-bf39-34d2118506e6-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:19.235498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.235476 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfcc37aa-99af-4a3c-bf39-34d2118506e6-kube-api-access-4jchd" pod "dfcc37aa-99af-4a3c-bf39-34d2118506e6" (UID: "dfcc37aa-99af-4a3c-bf39-34d2118506e6"). InnerVolumeSpecName "kube-api-access-4jchd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:19.334770 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.334735 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jchd\" (UniqueName: \"kubernetes.io/projected/dfcc37aa-99af-4a3c-bf39-34d2118506e6-kube-api-access-4jchd\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:19.430428 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.430396 4413 generic.go:356] "Generic (PLEG): container finished" podID="dfcc37aa-99af-4a3c-bf39-34d2118506e6" containerID="2b187a83eb5fb68a5cd7c5fe99b9b160d25a39d1a74393995f52fcbc04a24fa6" exitCode=0 Oct 30 12:25:19.430588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.430463 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx" Oct 30 12:25:19.430588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.430482 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx" event={"ID":"dfcc37aa-99af-4a3c-bf39-34d2118506e6","Type":"ContainerDied","Data":"2b187a83eb5fb68a5cd7c5fe99b9b160d25a39d1a74393995f52fcbc04a24fa6"} Oct 30 12:25:19.430588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.430546 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx" event={"ID":"dfcc37aa-99af-4a3c-bf39-34d2118506e6","Type":"ContainerDied","Data":"fa9cbcb80508d19c4638a5694511f03be4002d52e7d65c6317d737b463fe3ee5"} Oct 30 12:25:19.430588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.430568 4413 scope.go:122] "RemoveContainer" containerID="2b187a83eb5fb68a5cd7c5fe99b9b160d25a39d1a74393995f52fcbc04a24fa6" Oct 30 12:25:19.444575 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.443869 4413 scope.go:122] "RemoveContainer" containerID="2b187a83eb5fb68a5cd7c5fe99b9b160d25a39d1a74393995f52fcbc04a24fa6" Oct 30 12:25:19.446679 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:25:19.446654 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b187a83eb5fb68a5cd7c5fe99b9b160d25a39d1a74393995f52fcbc04a24fa6\": container with ID starting with 2b187a83eb5fb68a5cd7c5fe99b9b160d25a39d1a74393995f52fcbc04a24fa6 not found: ID does not exist" containerID="2b187a83eb5fb68a5cd7c5fe99b9b160d25a39d1a74393995f52fcbc04a24fa6" Oct 30 12:25:19.446767 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.446686 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b187a83eb5fb68a5cd7c5fe99b9b160d25a39d1a74393995f52fcbc04a24fa6"} err="failed to get container status \"2b187a83eb5fb68a5cd7c5fe99b9b160d25a39d1a74393995f52fcbc04a24fa6\": rpc error: code = NotFound desc = could not find container \"2b187a83eb5fb68a5cd7c5fe99b9b160d25a39d1a74393995f52fcbc04a24fa6\": container with ID starting with 2b187a83eb5fb68a5cd7c5fe99b9b160d25a39d1a74393995f52fcbc04a24fa6 not found: ID does not exist" Oct 30 12:25:19.481678 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.481654 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx"] Oct 30 12:25:19.487066 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:19.487042 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-951ad1-5b77ff48d6-bxcgx"] Oct 30 12:25:20.030874 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:20.030846 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:21.020734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:21.020692 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfcc37aa-99af-4a3c-bf39-34d2118506e6" path="/var/lib/kubelet/pods/dfcc37aa-99af-4a3c-bf39-34d2118506e6/volumes" Oct 30 12:25:21.447198 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:21.447165 4413 generic.go:356] "Generic (PLEG): container finished" podID="30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" containerID="fa61216a661768c2ffb1cfc935637ff581c445fce161d3b4f57521d3abf1d12f" exitCode=0 Oct 30 12:25:21.447646 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:21.447236 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" event={"ID":"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a","Type":"ContainerDied","Data":"fa61216a661768c2ffb1cfc935637ff581c445fce161d3b4f57521d3abf1d12f"} Oct 30 12:25:21.447696 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:21.447653 4413 scope.go:122] "RemoveContainer" containerID="fa61216a661768c2ffb1cfc935637ff581c445fce161d3b4f57521d3abf1d12f" Oct 30 12:25:21.449319 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:21.449290 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:25:21.449396 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:21.449366 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:25:21.449432 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:21.449400 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:25:21.689385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:21.689349 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:21.861132 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:21.861020 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.167572 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s"] Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168463 4413 cpu_manager.go:414] "containerMap: removing container" podUID="3a8cf876-a088-4dd2-9890-27772782da71" containerName="istio-proxy" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168480 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a8cf876-a088-4dd2-9890-27772782da71" containerName="istio-proxy" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168506 4413 cpu_manager.go:414] "containerMap: removing container" podUID="8eefa82e-f061-4c58-8fad-46c497d22c93" containerName="istio-proxy" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168513 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eefa82e-f061-4c58-8fad-46c497d22c93" containerName="istio-proxy" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168554 4413 cpu_manager.go:414] "containerMap: removing container" podUID="f7b5b44c-e86c-4990-a68b-1dbff2e42a46" containerName="istio-proxy" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168560 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b5b44c-e86c-4990-a68b-1dbff2e42a46" containerName="istio-proxy" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168570 4413 cpu_manager.go:414] "containerMap: removing container" podUID="dfcc37aa-99af-4a3c-bf39-34d2118506e6" containerName="payload-processing" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168577 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfcc37aa-99af-4a3c-bf39-34d2118506e6" containerName="payload-processing" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168590 4413 cpu_manager.go:414] "containerMap: removing container" podUID="1e53a452-5a36-4d78-9a14-35f6ebb715ba" containerName="istio-proxy" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168597 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e53a452-5a36-4d78-9a14-35f6ebb715ba" containerName="istio-proxy" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168621 4413 cpu_manager.go:414] "containerMap: removing container" podUID="7995a1f6-7cff-4b6a-b612-78b31bad73ba" containerName="istio-proxy" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168628 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7995a1f6-7cff-4b6a-b612-78b31bad73ba" containerName="istio-proxy" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168751 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a8cf876-a088-4dd2-9890-27772782da71" containerName="istio-proxy" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168764 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eefa82e-f061-4c58-8fad-46c497d22c93" containerName="istio-proxy" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168778 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b5b44c-e86c-4990-a68b-1dbff2e42a46" containerName="istio-proxy" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168785 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfcc37aa-99af-4a3c-bf39-34d2118506e6" containerName="payload-processing" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168795 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e53a452-5a36-4d78-9a14-35f6ebb715ba" containerName="istio-proxy" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168802 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eefa82e-f061-4c58-8fad-46c497d22c93" containerName="istio-proxy" Oct 30 12:25:22.170288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.168813 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="7995a1f6-7cff-4b6a-b612-78b31bad73ba" containerName="istio-proxy" Oct 30 12:25:22.182799 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.172081 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.186961 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.186938 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 12:25:22.186961 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.186959 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qvjrm\"" Oct 30 12:25:22.186961 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.186946 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 12:25:22.202470 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.202447 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s"] Oct 30 12:25:22.275748 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.275716 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.275892 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.275758 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.275951 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.275901 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.276084 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.276007 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/799fe64f-b3d4-43bf-b7ae-04833737b666-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.276153 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.276075 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.276189 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.276165 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc75m\" (UniqueName: \"kubernetes.io/projected/799fe64f-b3d4-43bf-b7ae-04833737b666-kube-api-access-hc75m\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.276224 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.276203 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.287554 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.287527 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 12:25:22.377844 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.377815 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.378032 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.377955 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.378032 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.377985 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.378032 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.378030 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.378170 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.378088 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/799fe64f-b3d4-43bf-b7ae-04833737b666-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.378170 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.378126 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.378263 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.378171 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc75m\" (UniqueName: \"kubernetes.io/projected/799fe64f-b3d4-43bf-b7ae-04833737b666-kube-api-access-hc75m\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.378688 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.378659 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.380706 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.380681 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.380848 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.380703 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.380975 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.380926 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.382330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.382310 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/799fe64f-b3d4-43bf-b7ae-04833737b666-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.402136 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.402109 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.402317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.402274 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc75m\" (UniqueName: \"kubernetes.io/projected/799fe64f-b3d4-43bf-b7ae-04833737b666-kube-api-access-hc75m\") pod \"istiod-openshift-gateway-7ddd9cfdc8-82z8s\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.456737 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.456647 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" event={"ID":"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a","Type":"ContainerStarted","Data":"aef235c22e3ec8d05b0db311520dcf2728817a67fc0981c3672d3786dfd41886"} Oct 30 12:25:22.509637 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.504607 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:22.832435 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:22.832399 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s"] Oct 30 12:25:22.833267 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:25:22.833212 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799fe64f_b3d4_43bf_b7ae_04833737b666.slice/crio-da0138d52bc45df7441a3bbc877092f07b4ff349d9c5920f9b3c24bc14c6c7ce WatchSource:0}: Error finding container da0138d52bc45df7441a3bbc877092f07b4ff349d9c5920f9b3c24bc14c6c7ce: Status 404 returned error can't find the container with id da0138d52bc45df7441a3bbc877092f07b4ff349d9c5920f9b3c24bc14c6c7ce Oct 30 12:25:23.215357 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:23.213558 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:25:23.216552 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:23.216524 4413 patch_prober.go:29] interesting pod/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 12:25:23.216699 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:23.216580 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" podUID="30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 12:25:23.316697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:23.316664 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:23.446311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:23.446260 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:23.470533 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:23.470426 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" event={"ID":"799fe64f-b3d4-43bf-b7ae-04833737b666","Type":"ContainerStarted","Data":"da0138d52bc45df7441a3bbc877092f07b4ff349d9c5920f9b3c24bc14c6c7ce"} Oct 30 12:25:24.222274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:24.221274 4413 patch_prober.go:29] interesting pod/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 12:25:24.222274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:24.221357 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" podUID="30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 12:25:25.216447 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:25.216409 4413 patch_prober.go:29] interesting pod/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 12:25:25.216927 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:25.216469 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" podUID="30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 12:25:25.244121 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:25.244085 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:26.215839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:26.215622 4413 patch_prober.go:29] interesting pod/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 12:25:26.215839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:26.215719 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" podUID="30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 12:25:26.263003 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:26.262970 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:25:26.263287 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:26.263036 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:25:26.529051 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:26.528953 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" event={"ID":"799fe64f-b3d4-43bf-b7ae-04833737b666","Type":"ContainerStarted","Data":"fb8c919024c0a744264bcb9c5e8b2ee24cea9c447d17b00e77b4c6153b1a655c"} Oct 30 12:25:26.529222 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:26.529091 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:26.621286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:26.619394 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" podStartSLOduration=1.193200377 podStartE2EDuration="4.619377581s" totalImagesPullingTime="3.426177204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:26.584137172 +0000 UTC m=+1879.693980054" watchObservedRunningTime="2026-10-30 12:25:26.619377581 +0000 UTC m=+1879.729220472" Oct 30 12:25:27.104825 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:27.104790 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:27.212799 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:27.212765 4413 patch_prober.go:29] interesting pod/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 12:25:27.213081 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:27.213037 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" podUID="30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 12:25:27.583750 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:27.583419 4413 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-82z8s container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:25:27.583750 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:27.583475 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" podUID="799fe64f-b3d4-43bf-b7ae-04833737b666" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:25:28.429206 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:28.429173 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:25:28.429895 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:28.429873 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:25:28.430836 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:28.430817 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:25:29.328176 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:29.328137 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:30.568063 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:30.568034 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:25:31.279762 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:31.279730 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:32.534715 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:32.534679 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:32.654397 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:32.654357 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:32.771700 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:32.771657 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:34.309545 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:34.309510 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:35.814497 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:25:35.814461 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:25:36.232932 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:36.232892 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:37.690072 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:37.690043 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n"] Oct 30 12:25:37.690563 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:37.690262 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" podUID="9f031740-a3c2-413b-9943-8d2d31f143a0" containerName="istio-proxy" containerID="cri-o://4f1f002f9cd2279eb36b3c6fa3cf6fb926ea09fecd5c86a7e688a97c5d99910c" gracePeriod=30 Oct 30 12:25:37.702469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:37.702436 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7"] Oct 30 12:25:37.702719 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:37.702694 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" podUID="682d04e3-a053-4d89-a812-d7669ed7c595" containerName="istio-proxy" containerID="cri-o://943bfc8852004c1793df262241811100d3a4d6c5ae3636cb2dba79f3477778c7" gracePeriod=30 Oct 30 12:25:37.705584 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:37.705558 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt"] Oct 30 12:25:37.705836 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:37.705806 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" podUID="b61e3e11-5a0b-4732-be67-e54717bc7909" containerName="istio-proxy" containerID="cri-o://639bf5d108664712e8dac6ec04e6da4387b00eb774eb27cf089ba016059ee5ed" gracePeriod=30 Oct 30 12:25:37.714550 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:37.714525 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg"] Oct 30 12:25:37.714747 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:37.714724 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" podUID="bc40e968-012f-4ca0-91c1-a67575f59dcd" containerName="istio-proxy" containerID="cri-o://75798be977ee300876bb90f56833a1cfbf4f27b13797a13cae6c83a84183b0f8" gracePeriod=30 Oct 30 12:25:38.205641 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:38.205604 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:40.171761 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:40.171729 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:42.277791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:42.277753 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:42.985621 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:42.985540 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:43.054716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.054360 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:25:43.055720 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.055696 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:25:43.105856 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.105830 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:25:43.111213 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.111188 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:25:43.165600 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.165570 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-workload-certs\") pod \"b61e3e11-5a0b-4732-be67-e54717bc7909\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " Oct 30 12:25:43.165791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.165635 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-workload-socket\") pod \"bc40e968-012f-4ca0-91c1-a67575f59dcd\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " Oct 30 12:25:43.165791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.165676 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-workload-socket\") pod \"682d04e3-a053-4d89-a812-d7669ed7c595\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " Oct 30 12:25:43.165791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.165724 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-data\") pod \"bc40e968-012f-4ca0-91c1-a67575f59dcd\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " Oct 30 12:25:43.165791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.165763 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-workload-certs\") pod \"682d04e3-a053-4d89-a812-d7669ed7c595\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " Oct 30 12:25:43.166012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.165798 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-workload-certs\") pod \"bc40e968-012f-4ca0-91c1-a67575f59dcd\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " Oct 30 12:25:43.166012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.165841 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-podinfo\") pod \"9f031740-a3c2-413b-9943-8d2d31f143a0\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " Oct 30 12:25:43.166012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.165867 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-workload-certs" pod "b61e3e11-5a0b-4732-be67-e54717bc7909" (UID: "b61e3e11-5a0b-4732-be67-e54717bc7909"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.166012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.165915 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-podinfo\") pod \"bc40e968-012f-4ca0-91c1-a67575f59dcd\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " Oct 30 12:25:43.166012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.165947 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-podinfo\") pod \"b61e3e11-5a0b-4732-be67-e54717bc7909\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " Oct 30 12:25:43.166012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.165976 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f031740-a3c2-413b-9943-8d2d31f143a0-kube-api-access-5d6k7\" (UniqueName: \"kubernetes.io/projected/9f031740-a3c2-413b-9943-8d2d31f143a0-kube-api-access-5d6k7\") pod \"9f031740-a3c2-413b-9943-8d2d31f143a0\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " Oct 30 12:25:43.166346 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166014 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc40e968-012f-4ca0-91c1-a67575f59dcd-kube-api-access-hg9t4\" (UniqueName: \"kubernetes.io/projected/bc40e968-012f-4ca0-91c1-a67575f59dcd-kube-api-access-hg9t4\") pod \"bc40e968-012f-4ca0-91c1-a67575f59dcd\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " Oct 30 12:25:43.166346 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166046 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-data" pod "bc40e968-012f-4ca0-91c1-a67575f59dcd" (UID: "bc40e968-012f-4ca0-91c1-a67575f59dcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.166346 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166044 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61e3e11-5a0b-4732-be67-e54717bc7909-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61e3e11-5a0b-4732-be67-e54717bc7909-istiod-ca-cert\") pod \"b61e3e11-5a0b-4732-be67-e54717bc7909\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " Oct 30 12:25:43.166346 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166149 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-workload-certs" pod "bc40e968-012f-4ca0-91c1-a67575f59dcd" (UID: "bc40e968-012f-4ca0-91c1-a67575f59dcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.166346 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166160 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-credential-socket\") pod \"682d04e3-a053-4d89-a812-d7669ed7c595\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " Oct 30 12:25:43.166346 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166206 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-workload-socket\") pod \"b61e3e11-5a0b-4732-be67-e54717bc7909\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " Oct 30 12:25:43.166346 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166273 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc40e968-012f-4ca0-91c1-a67575f59dcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc40e968-012f-4ca0-91c1-a67575f59dcd-istiod-ca-cert\") pod \"bc40e968-012f-4ca0-91c1-a67575f59dcd\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " Oct 30 12:25:43.166346 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166336 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-credential-socket\") pod \"b61e3e11-5a0b-4732-be67-e54717bc7909\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " Oct 30 12:25:43.166749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166394 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-credential-socket\") pod \"9f031740-a3c2-413b-9943-8d2d31f143a0\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " Oct 30 12:25:43.166749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166444 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61e3e11-5a0b-4732-be67-e54717bc7909-istiod-ca-cert" pod "b61e3e11-5a0b-4732-be67-e54717bc7909" (UID: "b61e3e11-5a0b-4732-be67-e54717bc7909"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:43.166749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166449 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f031740-a3c2-413b-9943-8d2d31f143a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f031740-a3c2-413b-9943-8d2d31f143a0-istiod-ca-cert\") pod \"9f031740-a3c2-413b-9943-8d2d31f143a0\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " Oct 30 12:25:43.166749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166503 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-istio-envoy\") pod \"682d04e3-a053-4d89-a812-d7669ed7c595\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " Oct 30 12:25:43.166749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166532 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-token\" (UniqueName: \"kubernetes.io/projected/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-token\") pod \"b61e3e11-5a0b-4732-be67-e54717bc7909\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " Oct 30 12:25:43.166749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166640 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-workload-certs" pod "682d04e3-a053-4d89-a812-d7669ed7c595" (UID: "682d04e3-a053-4d89-a812-d7669ed7c595"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.166749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166676 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-credential-socket\") pod \"bc40e968-012f-4ca0-91c1-a67575f59dcd\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " Oct 30 12:25:43.166749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166706 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/682d04e3-a053-4d89-a812-d7669ed7c595-istio-token\" (UniqueName: \"kubernetes.io/projected/682d04e3-a053-4d89-a812-d7669ed7c595-istio-token\") pod \"682d04e3-a053-4d89-a812-d7669ed7c595\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " Oct 30 12:25:43.167149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166772 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-envoy\") pod \"bc40e968-012f-4ca0-91c1-a67575f59dcd\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " Oct 30 12:25:43.167149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166842 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-workload-socket\") pod \"9f031740-a3c2-413b-9943-8d2d31f143a0\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " Oct 30 12:25:43.167149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166848 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f031740-a3c2-413b-9943-8d2d31f143a0-istiod-ca-cert" pod "9f031740-a3c2-413b-9943-8d2d31f143a0" (UID: "9f031740-a3c2-413b-9943-8d2d31f143a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:43.167149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166884 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/682d04e3-a053-4d89-a812-d7669ed7c595-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/682d04e3-a053-4d89-a812-d7669ed7c595-istio-podinfo\") pod \"682d04e3-a053-4d89-a812-d7669ed7c595\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " Oct 30 12:25:43.167149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166923 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-data\") pod \"b61e3e11-5a0b-4732-be67-e54717bc7909\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " Oct 30 12:25:43.167149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.166971 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-data\") pod \"9f031740-a3c2-413b-9943-8d2d31f143a0\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " Oct 30 12:25:43.167149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.167050 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-token\" (UniqueName: \"kubernetes.io/projected/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-token\") pod \"bc40e968-012f-4ca0-91c1-a67575f59dcd\" (UID: \"bc40e968-012f-4ca0-91c1-a67575f59dcd\") " Oct 30 12:25:43.167149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.167089 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/682d04e3-a053-4d89-a812-d7669ed7c595-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/682d04e3-a053-4d89-a812-d7669ed7c595-istiod-ca-cert\") pod \"682d04e3-a053-4d89-a812-d7669ed7c595\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " Oct 30 12:25:43.167149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.167133 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-credential-socket" pod "682d04e3-a053-4d89-a812-d7669ed7c595" (UID: "682d04e3-a053-4d89-a812-d7669ed7c595"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.167149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.167149 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-envoy\") pod \"9f031740-a3c2-413b-9943-8d2d31f143a0\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " Oct 30 12:25:43.167650 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.167200 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61e3e11-5a0b-4732-be67-e54717bc7909-kube-api-access-rh992\" (UniqueName: \"kubernetes.io/projected/b61e3e11-5a0b-4732-be67-e54717bc7909-kube-api-access-rh992\") pod \"b61e3e11-5a0b-4732-be67-e54717bc7909\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " Oct 30 12:25:43.167650 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.167240 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-envoy\") pod \"b61e3e11-5a0b-4732-be67-e54717bc7909\" (UID: \"b61e3e11-5a0b-4732-be67-e54717bc7909\") " Oct 30 12:25:43.167650 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.167296 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-workload-certs\") pod \"9f031740-a3c2-413b-9943-8d2d31f143a0\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " Oct 30 12:25:43.167650 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.167339 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-workload-socket" pod "b61e3e11-5a0b-4732-be67-e54717bc7909" (UID: "b61e3e11-5a0b-4732-be67-e54717bc7909"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.167650 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.167359 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-token\" (UniqueName: \"kubernetes.io/projected/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-token\") pod \"9f031740-a3c2-413b-9943-8d2d31f143a0\" (UID: \"9f031740-a3c2-413b-9943-8d2d31f143a0\") " Oct 30 12:25:43.167650 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.167404 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/682d04e3-a053-4d89-a812-d7669ed7c595-kube-api-access-bl4cp\" (UniqueName: \"kubernetes.io/projected/682d04e3-a053-4d89-a812-d7669ed7c595-kube-api-access-bl4cp\") pod \"682d04e3-a053-4d89-a812-d7669ed7c595\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " Oct 30 12:25:43.167650 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.167450 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-workload-socket" pod "682d04e3-a053-4d89-a812-d7669ed7c595" (UID: "682d04e3-a053-4d89-a812-d7669ed7c595"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.167650 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.167462 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-istio-data\") pod \"682d04e3-a053-4d89-a812-d7669ed7c595\" (UID: \"682d04e3-a053-4d89-a812-d7669ed7c595\") " Oct 30 12:25:43.168004 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.167661 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc40e968-012f-4ca0-91c1-a67575f59dcd-istiod-ca-cert" pod "bc40e968-012f-4ca0-91c1-a67575f59dcd" (UID: "bc40e968-012f-4ca0-91c1-a67575f59dcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:43.168004 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.167899 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-credential-socket" pod "b61e3e11-5a0b-4732-be67-e54717bc7909" (UID: "b61e3e11-5a0b-4732-be67-e54717bc7909"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.168766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.168385 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.168766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.168414 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.168766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.168431 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.168766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.168447 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.168766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.168460 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.168766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.168473 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61e3e11-5a0b-4732-be67-e54717bc7909-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.168766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.168487 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.168766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.168504 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.168766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.168518 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc40e968-012f-4ca0-91c1-a67575f59dcd-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.168766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.168534 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.168766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.168550 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f031740-a3c2-413b-9943-8d2d31f143a0-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.168766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.168754 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-podinfo" pod "9f031740-a3c2-413b-9943-8d2d31f143a0" (UID: "9f031740-a3c2-413b-9943-8d2d31f143a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:25:43.169502 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.169207 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:43.169617 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.169595 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-workload-socket" pod "9f031740-a3c2-413b-9943-8d2d31f143a0" (UID: "9f031740-a3c2-413b-9943-8d2d31f143a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.169798 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.169638 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-credential-socket" pod "9f031740-a3c2-413b-9943-8d2d31f143a0" (UID: "9f031740-a3c2-413b-9943-8d2d31f143a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.169897 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.169826 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-data" pod "b61e3e11-5a0b-4732-be67-e54717bc7909" (UID: "b61e3e11-5a0b-4732-be67-e54717bc7909"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.170170 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.170147 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-credential-socket" pod "bc40e968-012f-4ca0-91c1-a67575f59dcd" (UID: "bc40e968-012f-4ca0-91c1-a67575f59dcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.171676 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.171319 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-token" pod "b61e3e11-5a0b-4732-be67-e54717bc7909" (UID: "b61e3e11-5a0b-4732-be67-e54717bc7909"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:43.171676 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.171378 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-token" pod "bc40e968-012f-4ca0-91c1-a67575f59dcd" (UID: "bc40e968-012f-4ca0-91c1-a67575f59dcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:43.171676 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.171398 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-podinfo" pod "bc40e968-012f-4ca0-91c1-a67575f59dcd" (UID: "bc40e968-012f-4ca0-91c1-a67575f59dcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:25:43.171676 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.171620 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-workload-certs" pod "9f031740-a3c2-413b-9943-8d2d31f143a0" (UID: "9f031740-a3c2-413b-9943-8d2d31f143a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.172004 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.171780 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-data" pod "9f031740-a3c2-413b-9943-8d2d31f143a0" (UID: "9f031740-a3c2-413b-9943-8d2d31f143a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.172004 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.171907 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/682d04e3-a053-4d89-a812-d7669ed7c595-istiod-ca-cert" pod "682d04e3-a053-4d89-a812-d7669ed7c595" (UID: "682d04e3-a053-4d89-a812-d7669ed7c595"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:43.172984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.172954 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-istio-envoy" pod "682d04e3-a053-4d89-a812-d7669ed7c595" (UID: "682d04e3-a053-4d89-a812-d7669ed7c595"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.173233 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.173206 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-istio-data" pod "682d04e3-a053-4d89-a812-d7669ed7c595" (UID: "682d04e3-a053-4d89-a812-d7669ed7c595"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.173578 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.173554 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f031740-a3c2-413b-9943-8d2d31f143a0-kube-api-access-5d6k7" pod "9f031740-a3c2-413b-9943-8d2d31f143a0" (UID: "9f031740-a3c2-413b-9943-8d2d31f143a0"). InnerVolumeSpecName "kube-api-access-5d6k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:43.173692 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.173665 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-workload-socket" pod "bc40e968-012f-4ca0-91c1-a67575f59dcd" (UID: "bc40e968-012f-4ca0-91c1-a67575f59dcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.174840 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.174811 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-podinfo" pod "b61e3e11-5a0b-4732-be67-e54717bc7909" (UID: "b61e3e11-5a0b-4732-be67-e54717bc7909"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:25:43.176525 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.176502 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/682d04e3-a053-4d89-a812-d7669ed7c595-istio-token" pod "682d04e3-a053-4d89-a812-d7669ed7c595" (UID: "682d04e3-a053-4d89-a812-d7669ed7c595"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:43.176864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.176844 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/682d04e3-a053-4d89-a812-d7669ed7c595-kube-api-access-bl4cp" pod "682d04e3-a053-4d89-a812-d7669ed7c595" (UID: "682d04e3-a053-4d89-a812-d7669ed7c595"). InnerVolumeSpecName "kube-api-access-bl4cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:43.177110 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.177076 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-token" pod "9f031740-a3c2-413b-9943-8d2d31f143a0" (UID: "9f031740-a3c2-413b-9943-8d2d31f143a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:43.177544 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.177508 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/682d04e3-a053-4d89-a812-d7669ed7c595-istio-podinfo" pod "682d04e3-a053-4d89-a812-d7669ed7c595" (UID: "682d04e3-a053-4d89-a812-d7669ed7c595"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:25:43.177544 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.177510 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61e3e11-5a0b-4732-be67-e54717bc7909-kube-api-access-rh992" pod "b61e3e11-5a0b-4732-be67-e54717bc7909" (UID: "b61e3e11-5a0b-4732-be67-e54717bc7909"). InnerVolumeSpecName "kube-api-access-rh992". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:43.177647 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.177539 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc40e968-012f-4ca0-91c1-a67575f59dcd-kube-api-access-hg9t4" pod "bc40e968-012f-4ca0-91c1-a67575f59dcd" (UID: "bc40e968-012f-4ca0-91c1-a67575f59dcd"). InnerVolumeSpecName "kube-api-access-hg9t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:43.177647 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.177559 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-envoy" pod "9f031740-a3c2-413b-9943-8d2d31f143a0" (UID: "9f031740-a3c2-413b-9943-8d2d31f143a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.178509 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.178476 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-envoy" pod "bc40e968-012f-4ca0-91c1-a67575f59dcd" (UID: "bc40e968-012f-4ca0-91c1-a67575f59dcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.179548 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.179525 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-envoy" pod "b61e3e11-5a0b-4732-be67-e54717bc7909" (UID: "b61e3e11-5a0b-4732-be67-e54717bc7909"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:25:43.270400 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270327 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270400 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270354 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270400 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270363 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270400 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270371 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270400 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270379 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270400 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270388 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d6k7\" (UniqueName: \"kubernetes.io/projected/9f031740-a3c2-413b-9943-8d2d31f143a0-kube-api-access-5d6k7\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270400 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270397 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg9t4\" (UniqueName: \"kubernetes.io/projected/bc40e968-012f-4ca0-91c1-a67575f59dcd-kube-api-access-hg9t4\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270400 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270405 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270413 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/682d04e3-a053-4d89-a812-d7669ed7c595-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270421 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270428 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270435 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/682d04e3-a053-4d89-a812-d7669ed7c595-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270444 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270451 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270459 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/682d04e3-a053-4d89-a812-d7669ed7c595-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270466 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270473 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270480 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc40e968-012f-4ca0-91c1-a67575f59dcd-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270486 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/682d04e3-a053-4d89-a812-d7669ed7c595-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270494 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270501 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh992\" (UniqueName: \"kubernetes.io/projected/b61e3e11-5a0b-4732-be67-e54717bc7909-kube-api-access-rh992\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270508 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61e3e11-5a0b-4732-be67-e54717bc7909-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270516 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f031740-a3c2-413b-9943-8d2d31f143a0-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270523 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f031740-a3c2-413b-9943-8d2d31f143a0-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.270754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.270531 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl4cp\" (UniqueName: \"kubernetes.io/projected/682d04e3-a053-4d89-a812-d7669ed7c595-kube-api-access-bl4cp\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:43.451224 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.451193 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:43.567742 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.567614 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:43.675042 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.675013 4413 generic.go:356] "Generic (PLEG): container finished" podID="b61e3e11-5a0b-4732-be67-e54717bc7909" containerID="639bf5d108664712e8dac6ec04e6da4387b00eb774eb27cf089ba016059ee5ed" exitCode=0 Oct 30 12:25:43.675234 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.675115 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" event={"ID":"b61e3e11-5a0b-4732-be67-e54717bc7909","Type":"ContainerDied","Data":"639bf5d108664712e8dac6ec04e6da4387b00eb774eb27cf089ba016059ee5ed"} Oct 30 12:25:43.675519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.675141 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" Oct 30 12:25:43.675642 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.675166 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt" event={"ID":"b61e3e11-5a0b-4732-be67-e54717bc7909","Type":"ContainerDied","Data":"f9063dae8e5e7f575ba7e0aad158733013cdf1b33029e4101b524f1c26074fd8"} Oct 30 12:25:43.675709 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.675179 4413 scope.go:122] "RemoveContainer" containerID="639bf5d108664712e8dac6ec04e6da4387b00eb774eb27cf089ba016059ee5ed" Oct 30 12:25:43.677473 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.677452 4413 generic.go:356] "Generic (PLEG): container finished" podID="bc40e968-012f-4ca0-91c1-a67575f59dcd" containerID="75798be977ee300876bb90f56833a1cfbf4f27b13797a13cae6c83a84183b0f8" exitCode=0 Oct 30 12:25:43.677568 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.677499 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" Oct 30 12:25:43.677568 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.677539 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" event={"ID":"bc40e968-012f-4ca0-91c1-a67575f59dcd","Type":"ContainerDied","Data":"75798be977ee300876bb90f56833a1cfbf4f27b13797a13cae6c83a84183b0f8"} Oct 30 12:25:43.677698 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.677591 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg" event={"ID":"bc40e968-012f-4ca0-91c1-a67575f59dcd","Type":"ContainerDied","Data":"21d97e4e5dd8a5eefab9bb51823795c6620676d843f4072e9a0c2dac8a313b91"} Oct 30 12:25:43.679358 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.679328 4413 generic.go:356] "Generic (PLEG): container finished" podID="682d04e3-a053-4d89-a812-d7669ed7c595" containerID="943bfc8852004c1793df262241811100d3a4d6c5ae3636cb2dba79f3477778c7" exitCode=0 Oct 30 12:25:43.679753 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.679370 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" Oct 30 12:25:43.679753 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.679368 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" event={"ID":"682d04e3-a053-4d89-a812-d7669ed7c595","Type":"ContainerDied","Data":"943bfc8852004c1793df262241811100d3a4d6c5ae3636cb2dba79f3477778c7"} Oct 30 12:25:43.679753 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.679438 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7" event={"ID":"682d04e3-a053-4d89-a812-d7669ed7c595","Type":"ContainerDied","Data":"ddbabf0b86b0cf1917d60354bcd910b04485b26934643b2f061e2fd5ba22f7fe"} Oct 30 12:25:43.681178 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.681155 4413 generic.go:356] "Generic (PLEG): container finished" podID="9f031740-a3c2-413b-9943-8d2d31f143a0" containerID="4f1f002f9cd2279eb36b3c6fa3cf6fb926ea09fecd5c86a7e688a97c5d99910c" exitCode=0 Oct 30 12:25:43.681321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.681201 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" Oct 30 12:25:43.681321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.681276 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" event={"ID":"9f031740-a3c2-413b-9943-8d2d31f143a0","Type":"ContainerDied","Data":"4f1f002f9cd2279eb36b3c6fa3cf6fb926ea09fecd5c86a7e688a97c5d99910c"} Oct 30 12:25:43.681321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.681318 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n" event={"ID":"9f031740-a3c2-413b-9943-8d2d31f143a0","Type":"ContainerDied","Data":"ad6f28b212fa9cf35d208cecf26a735ab39db1ce025e0e2d905fbd82486660b2"} Oct 30 12:25:43.688882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.688865 4413 scope.go:122] "RemoveContainer" containerID="639bf5d108664712e8dac6ec04e6da4387b00eb774eb27cf089ba016059ee5ed" Oct 30 12:25:43.689178 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:25:43.689159 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639bf5d108664712e8dac6ec04e6da4387b00eb774eb27cf089ba016059ee5ed\": container with ID starting with 639bf5d108664712e8dac6ec04e6da4387b00eb774eb27cf089ba016059ee5ed not found: ID does not exist" containerID="639bf5d108664712e8dac6ec04e6da4387b00eb774eb27cf089ba016059ee5ed" Oct 30 12:25:43.689289 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.689186 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639bf5d108664712e8dac6ec04e6da4387b00eb774eb27cf089ba016059ee5ed"} err="failed to get container status \"639bf5d108664712e8dac6ec04e6da4387b00eb774eb27cf089ba016059ee5ed\": rpc error: code = NotFound desc = could not find container \"639bf5d108664712e8dac6ec04e6da4387b00eb774eb27cf089ba016059ee5ed\": container with ID starting with 639bf5d108664712e8dac6ec04e6da4387b00eb774eb27cf089ba016059ee5ed not found: ID does not exist" Oct 30 12:25:43.689289 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.689203 4413 scope.go:122] "RemoveContainer" containerID="75798be977ee300876bb90f56833a1cfbf4f27b13797a13cae6c83a84183b0f8" Oct 30 12:25:43.700153 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.700135 4413 scope.go:122] "RemoveContainer" containerID="afd4b9aec95564159b5c65cb2951672a68c9f6cdaa58bf5f7ef9c02724aa7d42" Oct 30 12:25:43.746353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.743509 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt"] Oct 30 12:25:43.746353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.744827 4413 scope.go:122] "RemoveContainer" containerID="75798be977ee300876bb90f56833a1cfbf4f27b13797a13cae6c83a84183b0f8" Oct 30 12:25:43.750068 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.750027 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-ldqkt"] Oct 30 12:25:43.752632 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:25:43.752581 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75798be977ee300876bb90f56833a1cfbf4f27b13797a13cae6c83a84183b0f8\": container with ID starting with 75798be977ee300876bb90f56833a1cfbf4f27b13797a13cae6c83a84183b0f8 not found: ID does not exist" containerID="75798be977ee300876bb90f56833a1cfbf4f27b13797a13cae6c83a84183b0f8" Oct 30 12:25:43.752805 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.752633 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75798be977ee300876bb90f56833a1cfbf4f27b13797a13cae6c83a84183b0f8"} err="failed to get container status \"75798be977ee300876bb90f56833a1cfbf4f27b13797a13cae6c83a84183b0f8\": rpc error: code = NotFound desc = could not find container \"75798be977ee300876bb90f56833a1cfbf4f27b13797a13cae6c83a84183b0f8\": container with ID starting with 75798be977ee300876bb90f56833a1cfbf4f27b13797a13cae6c83a84183b0f8 not found: ID does not exist" Oct 30 12:25:43.752805 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.752665 4413 scope.go:122] "RemoveContainer" containerID="afd4b9aec95564159b5c65cb2951672a68c9f6cdaa58bf5f7ef9c02724aa7d42" Oct 30 12:25:43.753324 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:25:43.753290 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd4b9aec95564159b5c65cb2951672a68c9f6cdaa58bf5f7ef9c02724aa7d42\": container with ID starting with afd4b9aec95564159b5c65cb2951672a68c9f6cdaa58bf5f7ef9c02724aa7d42 not found: ID does not exist" containerID="afd4b9aec95564159b5c65cb2951672a68c9f6cdaa58bf5f7ef9c02724aa7d42" Oct 30 12:25:43.753424 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.753330 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd4b9aec95564159b5c65cb2951672a68c9f6cdaa58bf5f7ef9c02724aa7d42"} err="failed to get container status \"afd4b9aec95564159b5c65cb2951672a68c9f6cdaa58bf5f7ef9c02724aa7d42\": rpc error: code = NotFound desc = could not find container \"afd4b9aec95564159b5c65cb2951672a68c9f6cdaa58bf5f7ef9c02724aa7d42\": container with ID starting with afd4b9aec95564159b5c65cb2951672a68c9f6cdaa58bf5f7ef9c02724aa7d42 not found: ID does not exist" Oct 30 12:25:43.753424 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.753351 4413 scope.go:122] "RemoveContainer" containerID="943bfc8852004c1793df262241811100d3a4d6c5ae3636cb2dba79f3477778c7" Oct 30 12:25:43.766381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.766348 4413 scope.go:122] "RemoveContainer" containerID="943bfc8852004c1793df262241811100d3a4d6c5ae3636cb2dba79f3477778c7" Oct 30 12:25:43.766890 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:25:43.766860 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"943bfc8852004c1793df262241811100d3a4d6c5ae3636cb2dba79f3477778c7\": container with ID starting with 943bfc8852004c1793df262241811100d3a4d6c5ae3636cb2dba79f3477778c7 not found: ID does not exist" containerID="943bfc8852004c1793df262241811100d3a4d6c5ae3636cb2dba79f3477778c7" Oct 30 12:25:43.767072 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.766902 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"943bfc8852004c1793df262241811100d3a4d6c5ae3636cb2dba79f3477778c7"} err="failed to get container status \"943bfc8852004c1793df262241811100d3a4d6c5ae3636cb2dba79f3477778c7\": rpc error: code = NotFound desc = could not find container \"943bfc8852004c1793df262241811100d3a4d6c5ae3636cb2dba79f3477778c7\": container with ID starting with 943bfc8852004c1793df262241811100d3a4d6c5ae3636cb2dba79f3477778c7 not found: ID does not exist" Oct 30 12:25:43.767072 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.766921 4413 scope.go:122] "RemoveContainer" containerID="4f1f002f9cd2279eb36b3c6fa3cf6fb926ea09fecd5c86a7e688a97c5d99910c" Oct 30 12:25:43.780441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.780414 4413 scope.go:122] "RemoveContainer" containerID="4f1f002f9cd2279eb36b3c6fa3cf6fb926ea09fecd5c86a7e688a97c5d99910c" Oct 30 12:25:43.780770 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:25:43.780750 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f1f002f9cd2279eb36b3c6fa3cf6fb926ea09fecd5c86a7e688a97c5d99910c\": container with ID starting with 4f1f002f9cd2279eb36b3c6fa3cf6fb926ea09fecd5c86a7e688a97c5d99910c not found: ID does not exist" containerID="4f1f002f9cd2279eb36b3c6fa3cf6fb926ea09fecd5c86a7e688a97c5d99910c" Oct 30 12:25:43.780855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.780780 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1f002f9cd2279eb36b3c6fa3cf6fb926ea09fecd5c86a7e688a97c5d99910c"} err="failed to get container status \"4f1f002f9cd2279eb36b3c6fa3cf6fb926ea09fecd5c86a7e688a97c5d99910c\": rpc error: code = NotFound desc = could not find container \"4f1f002f9cd2279eb36b3c6fa3cf6fb926ea09fecd5c86a7e688a97c5d99910c\": container with ID starting with 4f1f002f9cd2279eb36b3c6fa3cf6fb926ea09fecd5c86a7e688a97c5d99910c not found: ID does not exist" Oct 30 12:25:43.812135 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.812093 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7"] Oct 30 12:25:43.815194 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.815174 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-fnpm7"] Oct 30 12:25:43.848504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.848429 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg"] Oct 30 12:25:43.859412 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.859383 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-c8nxg"] Oct 30 12:25:43.893097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.893024 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n"] Oct 30 12:25:43.897558 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:43.897531 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-512ec5-openshift-default-64bf6cc76c-h7m6n"] Oct 30 12:25:44.097902 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.097865 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:44.398765 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.398735 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g"] Oct 30 12:25:44.400455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.400431 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:44.442674 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.442651 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm"] Oct 30 12:25:44.539070 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.539040 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx"] Oct 30 12:25:44.539896 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.539875 4413 cpu_manager.go:414] "containerMap: removing container" podUID="bc40e968-012f-4ca0-91c1-a67575f59dcd" containerName="istio-proxy" Oct 30 12:25:44.540010 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.539897 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc40e968-012f-4ca0-91c1-a67575f59dcd" containerName="istio-proxy" Oct 30 12:25:44.540010 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.539917 4413 cpu_manager.go:414] "containerMap: removing container" podUID="8eefa82e-f061-4c58-8fad-46c497d22c93" containerName="istio-proxy" Oct 30 12:25:44.540010 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.539924 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eefa82e-f061-4c58-8fad-46c497d22c93" containerName="istio-proxy" Oct 30 12:25:44.540010 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.539966 4413 cpu_manager.go:414] "containerMap: removing container" podUID="b61e3e11-5a0b-4732-be67-e54717bc7909" containerName="istio-proxy" Oct 30 12:25:44.540010 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.539973 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61e3e11-5a0b-4732-be67-e54717bc7909" containerName="istio-proxy" Oct 30 12:25:44.540010 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.539990 4413 cpu_manager.go:414] "containerMap: removing container" podUID="bc40e968-012f-4ca0-91c1-a67575f59dcd" containerName="istio-proxy" Oct 30 12:25:44.540010 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.539996 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc40e968-012f-4ca0-91c1-a67575f59dcd" containerName="istio-proxy" Oct 30 12:25:44.540398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.540037 4413 cpu_manager.go:414] "containerMap: removing container" podUID="682d04e3-a053-4d89-a812-d7669ed7c595" containerName="istio-proxy" Oct 30 12:25:44.540398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.540045 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="682d04e3-a053-4d89-a812-d7669ed7c595" containerName="istio-proxy" Oct 30 12:25:44.540398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.540057 4413 cpu_manager.go:414] "containerMap: removing container" podUID="9f031740-a3c2-413b-9943-8d2d31f143a0" containerName="istio-proxy" Oct 30 12:25:44.540398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.540063 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f031740-a3c2-413b-9943-8d2d31f143a0" containerName="istio-proxy" Oct 30 12:25:44.540398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.540200 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61e3e11-5a0b-4732-be67-e54717bc7909" containerName="istio-proxy" Oct 30 12:25:44.540398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.540213 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc40e968-012f-4ca0-91c1-a67575f59dcd" containerName="istio-proxy" Oct 30 12:25:44.540398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.540227 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="682d04e3-a053-4d89-a812-d7669ed7c595" containerName="istio-proxy" Oct 30 12:25:44.540398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.540237 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f031740-a3c2-413b-9943-8d2d31f143a0" containerName="istio-proxy" Oct 30 12:25:44.540398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.540264 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc40e968-012f-4ca0-91c1-a67575f59dcd" containerName="istio-proxy" Oct 30 12:25:44.543388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.543368 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.561926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.561905 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-431bf6-openshift-default-dockercfg-frkkm\"" Oct 30 12:25:44.569152 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.569119 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx"] Oct 30 12:25:44.574618 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.574596 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:44.691069 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.690974 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:44.692662 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.692637 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5235ef15-0da6-4593-85a6-8d453b249064-istio-podinfo\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.692796 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.692712 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5235ef15-0da6-4593-85a6-8d453b249064-istio-token\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.692796 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.692779 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-istio-envoy\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.692877 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.692808 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-workload-certs\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.692877 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.692837 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-workload-socket\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.692949 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.692943 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-istio-data\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.692991 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.692972 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-credential-socket\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.693027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.692999 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5235ef15-0da6-4593-85a6-8d453b249064-istiod-ca-cert\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.693027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.693015 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9ft9\" (UniqueName: \"kubernetes.io/projected/5235ef15-0da6-4593-85a6-8d453b249064-kube-api-access-c9ft9\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.794735 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.794692 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-workload-socket\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.794964 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.794889 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-istio-data\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.795043 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.794959 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-credential-socket\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.795043 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.795007 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5235ef15-0da6-4593-85a6-8d453b249064-istiod-ca-cert\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.795043 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.795033 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9ft9\" (UniqueName: \"kubernetes.io/projected/5235ef15-0da6-4593-85a6-8d453b249064-kube-api-access-c9ft9\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.795202 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.795117 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5235ef15-0da6-4593-85a6-8d453b249064-istio-podinfo\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.795202 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.795164 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5235ef15-0da6-4593-85a6-8d453b249064-istio-token\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.795327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.795225 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-istio-envoy\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.795327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.795308 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-workload-certs\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.795833 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.795803 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-workload-certs\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.796265 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.796223 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-credential-socket\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.796636 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.796613 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-istio-data\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.796727 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.796664 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-workload-socket\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.796727 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.796717 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5235ef15-0da6-4593-85a6-8d453b249064-istiod-ca-cert\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.799628 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.799600 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5235ef15-0da6-4593-85a6-8d453b249064-istio-podinfo\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.801754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.801729 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-istio-envoy\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.819786 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.819753 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5235ef15-0da6-4593-85a6-8d453b249064-istio-token\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.819945 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.819848 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9ft9\" (UniqueName: \"kubernetes.io/projected/5235ef15-0da6-4593-85a6-8d453b249064-kube-api-access-c9ft9\") pod \"e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:44.864922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:44.864887 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:45.020892 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:45.020806 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="682d04e3-a053-4d89-a812-d7669ed7c595" path="/var/lib/kubelet/pods/682d04e3-a053-4d89-a812-d7669ed7c595/volumes" Oct 30 12:25:45.021435 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:45.021419 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f031740-a3c2-413b-9943-8d2d31f143a0" path="/var/lib/kubelet/pods/9f031740-a3c2-413b-9943-8d2d31f143a0/volumes" Oct 30 12:25:45.021923 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:45.021909 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61e3e11-5a0b-4732-be67-e54717bc7909" path="/var/lib/kubelet/pods/b61e3e11-5a0b-4732-be67-e54717bc7909/volumes" Oct 30 12:25:45.022688 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:45.022670 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc40e968-012f-4ca0-91c1-a67575f59dcd" path="/var/lib/kubelet/pods/bc40e968-012f-4ca0-91c1-a67575f59dcd/volumes" Oct 30 12:25:45.136265 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:45.136219 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:45.194469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:45.194446 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx"] Oct 30 12:25:45.195474 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:25:45.195445 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5235ef15_0da6_4593_85a6_8d453b249064.slice/crio-9a1cb00133a0bbc93f898f38c63aa521c7699dd48f8529073a3b05fcfc9fe21b WatchSource:0}: Error finding container 9a1cb00133a0bbc93f898f38c63aa521c7699dd48f8529073a3b05fcfc9fe21b: Status 404 returned error can't find the container with id 9a1cb00133a0bbc93f898f38c63aa521c7699dd48f8529073a3b05fcfc9fe21b Oct 30 12:25:45.198326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:45.198288 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:25:45.198425 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:45.198365 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:25:45.198425 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:45.198409 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:25:45.710504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:45.710465 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" event={"ID":"5235ef15-0da6-4593-85a6-8d453b249064","Type":"ContainerStarted","Data":"c4d0454bb4d05d0982eb72d09adb224e1b67f6f1df01d0a62beb452ff10621e3"} Oct 30 12:25:45.710962 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:45.710532 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" event={"ID":"5235ef15-0da6-4593-85a6-8d453b249064","Type":"ContainerStarted","Data":"9a1cb00133a0bbc93f898f38c63aa521c7699dd48f8529073a3b05fcfc9fe21b"} Oct 30 12:25:45.758623 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:45.758553 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" podStartSLOduration=1.758530716 podStartE2EDuration="1.758530716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:45.755117519 +0000 UTC m=+1898.864960401" watchObservedRunningTime="2026-10-30 12:25:45.758530716 +0000 UTC m=+1898.868373598" Oct 30 12:25:45.763039 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:45.763006 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:45.865157 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:45.865127 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:45.875738 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:45.875699 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:46.518076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:46.518046 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:46.872202 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:46.872126 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:47.726409 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:47.726380 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:47.727713 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:47.727681 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:25:48.398729 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:48.398695 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:48.579589 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:48.579553 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:48.737737 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:48.737627 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:49.399442 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.399397 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g" podUID="29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe" containerName="payload-processing" containerID="cri-o://cdc1196e8caca77d4b06db0c8d0095f9fa73c0402b9594086450f8d9530ea211" gracePeriod=25 Oct 30 12:25:49.442900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.442866 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm" podUID="3369323e-156f-44da-b214-6d8e63a97956" containerName="payload-pre-processing" containerID="cri-o://4db8aae06533eddafb1f4fa529e1cb1a7da6661575669fa1f6b7802bd7a065cb" gracePeriod=25 Oct 30 12:25:49.719607 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.719580 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g" Oct 30 12:25:49.742824 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.742789 4413 generic.go:356] "Generic (PLEG): container finished" podID="3369323e-156f-44da-b214-6d8e63a97956" containerID="4db8aae06533eddafb1f4fa529e1cb1a7da6661575669fa1f6b7802bd7a065cb" exitCode=0 Oct 30 12:25:49.742975 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.742858 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm" event={"ID":"3369323e-156f-44da-b214-6d8e63a97956","Type":"ContainerDied","Data":"4db8aae06533eddafb1f4fa529e1cb1a7da6661575669fa1f6b7802bd7a065cb"} Oct 30 12:25:49.744526 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.744498 4413 generic.go:356] "Generic (PLEG): container finished" podID="29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe" containerID="cdc1196e8caca77d4b06db0c8d0095f9fa73c0402b9594086450f8d9530ea211" exitCode=0 Oct 30 12:25:49.744675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.744552 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g" Oct 30 12:25:49.744675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.744583 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g" event={"ID":"29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe","Type":"ContainerDied","Data":"cdc1196e8caca77d4b06db0c8d0095f9fa73c0402b9594086450f8d9530ea211"} Oct 30 12:25:49.744675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.744642 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g" event={"ID":"29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe","Type":"ContainerDied","Data":"1a0f24bc7907f50efb2ba9f57faac25a060d3f58e046107f2388c508003ea893"} Oct 30 12:25:49.744675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.744672 4413 scope.go:122] "RemoveContainer" containerID="cdc1196e8caca77d4b06db0c8d0095f9fa73c0402b9594086450f8d9530ea211" Oct 30 12:25:49.757697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.757664 4413 scope.go:122] "RemoveContainer" containerID="cdc1196e8caca77d4b06db0c8d0095f9fa73c0402b9594086450f8d9530ea211" Oct 30 12:25:49.758234 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:25:49.758162 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdc1196e8caca77d4b06db0c8d0095f9fa73c0402b9594086450f8d9530ea211\": container with ID starting with cdc1196e8caca77d4b06db0c8d0095f9fa73c0402b9594086450f8d9530ea211 not found: ID does not exist" containerID="cdc1196e8caca77d4b06db0c8d0095f9fa73c0402b9594086450f8d9530ea211" Oct 30 12:25:49.758234 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.758209 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc1196e8caca77d4b06db0c8d0095f9fa73c0402b9594086450f8d9530ea211"} err="failed to get container status \"cdc1196e8caca77d4b06db0c8d0095f9fa73c0402b9594086450f8d9530ea211\": rpc error: code = NotFound desc = could not find container \"cdc1196e8caca77d4b06db0c8d0095f9fa73c0402b9594086450f8d9530ea211\": container with ID starting with cdc1196e8caca77d4b06db0c8d0095f9fa73c0402b9594086450f8d9530ea211 not found: ID does not exist" Oct 30 12:25:49.760481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.760458 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm" Oct 30 12:25:49.875884 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.875859 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe-plugins-config-volume\") pod \"29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe\" (UID: \"29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe\") " Oct 30 12:25:49.876054 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.875989 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe-kube-api-access-qxghs\" (UniqueName: \"kubernetes.io/projected/29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe-kube-api-access-qxghs\") pod \"29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe\" (UID: \"29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe\") " Oct 30 12:25:49.876204 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.876187 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3369323e-156f-44da-b214-6d8e63a97956-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3369323e-156f-44da-b214-6d8e63a97956-plugins-config-volume\") pod \"3369323e-156f-44da-b214-6d8e63a97956\" (UID: \"3369323e-156f-44da-b214-6d8e63a97956\") " Oct 30 12:25:49.876243 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.876213 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe-plugins-config-volume" pod "29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe" (UID: "29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:49.876297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.876280 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3369323e-156f-44da-b214-6d8e63a97956-kube-api-access-xvjzw\" (UniqueName: \"kubernetes.io/projected/3369323e-156f-44da-b214-6d8e63a97956-kube-api-access-xvjzw\") pod \"3369323e-156f-44da-b214-6d8e63a97956\" (UID: \"3369323e-156f-44da-b214-6d8e63a97956\") " Oct 30 12:25:49.876584 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.876550 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3369323e-156f-44da-b214-6d8e63a97956-plugins-config-volume" pod "3369323e-156f-44da-b214-6d8e63a97956" (UID: "3369323e-156f-44da-b214-6d8e63a97956"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:25:49.876908 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.876886 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3369323e-156f-44da-b214-6d8e63a97956-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:49.877033 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.876912 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:49.878358 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.878335 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe-kube-api-access-qxghs" pod "29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe" (UID: "29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe"). InnerVolumeSpecName "kube-api-access-qxghs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:49.878581 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.878566 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3369323e-156f-44da-b214-6d8e63a97956-kube-api-access-xvjzw" pod "3369323e-156f-44da-b214-6d8e63a97956" (UID: "3369323e-156f-44da-b214-6d8e63a97956"). InnerVolumeSpecName "kube-api-access-xvjzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:25:49.977852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.977783 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxghs\" (UniqueName: \"kubernetes.io/projected/29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe-kube-api-access-qxghs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:49.977852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:49.977811 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvjzw\" (UniqueName: \"kubernetes.io/projected/3369323e-156f-44da-b214-6d8e63a97956-kube-api-access-xvjzw\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:25:50.012436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.012405 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:50.091816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.091789 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g"] Oct 30 12:25:50.094506 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.094474 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-922643-564f9bdf4-c9v9g"] Oct 30 12:25:50.183481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.183342 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:50.187237 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.187212 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv"] Oct 30 12:25:50.187864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.187846 4413 cpu_manager.go:414] "containerMap: removing container" podUID="29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe" containerName="payload-processing" Oct 30 12:25:50.187959 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.187866 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe" containerName="payload-processing" Oct 30 12:25:50.187959 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.187914 4413 cpu_manager.go:414] "containerMap: removing container" podUID="3369323e-156f-44da-b214-6d8e63a97956" containerName="payload-pre-processing" Oct 30 12:25:50.187959 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.187923 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3369323e-156f-44da-b214-6d8e63a97956" containerName="payload-pre-processing" Oct 30 12:25:50.188122 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.188078 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe" containerName="payload-processing" Oct 30 12:25:50.188122 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.188100 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="3369323e-156f-44da-b214-6d8e63a97956" containerName="payload-pre-processing" Oct 30 12:25:50.191885 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.191868 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv" Oct 30 12:25:50.206379 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.206354 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-431bf6\"" Oct 30 12:25:50.206379 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.206368 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-431bf6-dockercfg-mcjgz\"" Oct 30 12:25:50.209338 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.209307 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv"] Oct 30 12:25:50.295322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.285586 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24"] Oct 30 12:25:50.299271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.296344 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24" Oct 30 12:25:50.342273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.328808 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24"] Oct 30 12:25:50.384653 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.384616 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edf395c4-4970-4487-9952-fab1b6c07f6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv\" (UID: \"edf395c4-4970-4487-9952-fab1b6c07f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv" Oct 30 12:25:50.384833 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.384817 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwhs5\" (UniqueName: \"kubernetes.io/projected/edf395c4-4970-4487-9952-fab1b6c07f6a-kube-api-access-pwhs5\") pod \"payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv\" (UID: \"edf395c4-4970-4487-9952-fab1b6c07f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv" Oct 30 12:25:50.485839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.485802 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edf395c4-4970-4487-9952-fab1b6c07f6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv\" (UID: \"edf395c4-4970-4487-9952-fab1b6c07f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv" Oct 30 12:25:50.486280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.485886 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqc7h\" (UniqueName: \"kubernetes.io/projected/d0994ac6-f047-4b32-ad6b-acad3336a059-kube-api-access-dqc7h\") pod \"payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24\" (UID: \"d0994ac6-f047-4b32-ad6b-acad3336a059\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24" Oct 30 12:25:50.486280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.486087 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwhs5\" (UniqueName: \"kubernetes.io/projected/edf395c4-4970-4487-9952-fab1b6c07f6a-kube-api-access-pwhs5\") pod \"payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv\" (UID: \"edf395c4-4970-4487-9952-fab1b6c07f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv" Oct 30 12:25:50.486280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.486131 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0994ac6-f047-4b32-ad6b-acad3336a059-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24\" (UID: \"d0994ac6-f047-4b32-ad6b-acad3336a059\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24" Oct 30 12:25:50.486675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.486647 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edf395c4-4970-4487-9952-fab1b6c07f6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv\" (UID: \"edf395c4-4970-4487-9952-fab1b6c07f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv" Oct 30 12:25:50.510682 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.510649 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwhs5\" (UniqueName: \"kubernetes.io/projected/edf395c4-4970-4487-9952-fab1b6c07f6a-kube-api-access-pwhs5\") pod \"payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv\" (UID: \"edf395c4-4970-4487-9952-fab1b6c07f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv" Oct 30 12:25:50.587469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.587394 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqc7h\" (UniqueName: \"kubernetes.io/projected/d0994ac6-f047-4b32-ad6b-acad3336a059-kube-api-access-dqc7h\") pod \"payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24\" (UID: \"d0994ac6-f047-4b32-ad6b-acad3336a059\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24" Oct 30 12:25:50.587617 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.587497 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0994ac6-f047-4b32-ad6b-acad3336a059-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24\" (UID: \"d0994ac6-f047-4b32-ad6b-acad3336a059\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24" Oct 30 12:25:50.588155 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.588136 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0994ac6-f047-4b32-ad6b-acad3336a059-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24\" (UID: \"d0994ac6-f047-4b32-ad6b-acad3336a059\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24" Oct 30 12:25:50.592106 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.592080 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:50.611241 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.611214 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqc7h\" (UniqueName: \"kubernetes.io/projected/d0994ac6-f047-4b32-ad6b-acad3336a059-kube-api-access-dqc7h\") pod \"payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24\" (UID: \"d0994ac6-f047-4b32-ad6b-acad3336a059\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24" Oct 30 12:25:50.651487 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.651453 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24" Oct 30 12:25:50.753840 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.753814 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm" Oct 30 12:25:50.754046 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.753808 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm" event={"ID":"3369323e-156f-44da-b214-6d8e63a97956","Type":"ContainerDied","Data":"2135df74a5ec392d967ce9b4f328c777ec64c0253962ede44ebfbc276efa94c7"} Oct 30 12:25:50.754046 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.753958 4413 scope.go:122] "RemoveContainer" containerID="4db8aae06533eddafb1f4fa529e1cb1a7da6661575669fa1f6b7802bd7a065cb" Oct 30 12:25:50.805510 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.805470 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm"] Oct 30 12:25:50.805863 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.805699 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv" Oct 30 12:25:50.809748 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.809724 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-922643-57d7dd5775-cv2tm"] Oct 30 12:25:50.957326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:50.957304 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24"] Oct 30 12:25:50.958615 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:25:50.958586 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0994ac6_f047_4b32_ad6b_acad3336a059.slice/crio-b101572ec1b876562d737e1391bb4194ceb78aaddaa2936389bb17a377e9d4dd WatchSource:0}: Error finding container b101572ec1b876562d737e1391bb4194ceb78aaddaa2936389bb17a377e9d4dd: Status 404 returned error can't find the container with id b101572ec1b876562d737e1391bb4194ceb78aaddaa2936389bb17a377e9d4dd Oct 30 12:25:51.022370 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.022343 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe" path="/var/lib/kubelet/pods/29aaf5f4-e6f6-4b96-8c71-8b13a2cacebe/volumes" Oct 30 12:25:51.022859 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.022841 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3369323e-156f-44da-b214-6d8e63a97956" path="/var/lib/kubelet/pods/3369323e-156f-44da-b214-6d8e63a97956/volumes" Oct 30 12:25:51.119560 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.119530 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv"] Oct 30 12:25:51.219587 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.219549 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:51.649852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.649809 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp"] Oct 30 12:25:51.654394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.654364 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.674763 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.674717 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-266486-openshift-default-dockercfg-z799c\"" Oct 30 12:25:51.687621 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.687574 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp"] Oct 30 12:25:51.703579 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.703545 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-workload-socket\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.703579 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.703583 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-envoy\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.703853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.703669 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-token\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.703853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.703690 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e658f2-c95e-40fc-9602-ac8271892d05-istiod-ca-cert\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.703853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.703764 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-workload-certs\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.704039 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.703875 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-credential-socket\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.704039 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.703909 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcgqj\" (UniqueName: \"kubernetes.io/projected/f1e658f2-c95e-40fc-9602-ac8271892d05-kube-api-access-bcgqj\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.704039 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.704006 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-podinfo\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.704205 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.704057 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-data\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.724067 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.724042 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:51.762542 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.762513 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv" event={"ID":"edf395c4-4970-4487-9952-fab1b6c07f6a","Type":"ContainerStarted","Data":"16a56cf34935c864857d9dd0e0ea03c695a9b87c7c87e03c9d45f6d2292e943b"} Oct 30 12:25:51.762703 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.762561 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv" event={"ID":"edf395c4-4970-4487-9952-fab1b6c07f6a","Type":"ContainerStarted","Data":"76844d587545dd678ac5cff2f4c99a241f03c7699a4dad2c3f1cc8ba2aaf03fd"} Oct 30 12:25:51.762703 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.762609 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv" Oct 30 12:25:51.764396 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.764369 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24" event={"ID":"d0994ac6-f047-4b32-ad6b-acad3336a059","Type":"ContainerStarted","Data":"0aab0c35802d9bac23a1a3630a1528713249714f82e8ecf05d0f7930fd2cc79d"} Oct 30 12:25:51.764517 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.764410 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24" event={"ID":"d0994ac6-f047-4b32-ad6b-acad3336a059","Type":"ContainerStarted","Data":"b101572ec1b876562d737e1391bb4194ceb78aaddaa2936389bb17a377e9d4dd"} Oct 30 12:25:51.764517 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.764437 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24" Oct 30 12:25:51.804780 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.804719 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv" podStartSLOduration=1.804700086 podStartE2EDuration="1.804700086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:51.801469002 +0000 UTC m=+1904.911311883" watchObservedRunningTime="2026-10-30 12:25:51.804700086 +0000 UTC m=+1904.914542971" Oct 30 12:25:51.804970 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.804804 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e658f2-c95e-40fc-9602-ac8271892d05-istiod-ca-cert\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.804970 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.804836 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-workload-certs\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.804970 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.804905 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-credential-socket\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.804970 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.804931 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcgqj\" (UniqueName: \"kubernetes.io/projected/f1e658f2-c95e-40fc-9602-ac8271892d05-kube-api-access-bcgqj\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.805198 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.805029 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-podinfo\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.805198 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.805063 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-data\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.805198 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.805166 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-workload-socket\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.805394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.805196 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-envoy\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.805394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.805315 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-token\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.805394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.805342 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-workload-certs\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.805816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.805793 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-data\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.805926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.805853 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-credential-socket\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.805994 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.805963 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-workload-socket\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.806832 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.806776 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e658f2-c95e-40fc-9602-ac8271892d05-istiod-ca-cert\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.808603 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.808584 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-envoy\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.809084 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.809065 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-podinfo\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.887278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.877807 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:51.887278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.880203 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24" podStartSLOduration=1.880187013 podStartE2EDuration="1.880187013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:51.860165994 +0000 UTC m=+1904.970008876" watchObservedRunningTime="2026-10-30 12:25:51.880187013 +0000 UTC m=+1904.990029896" Oct 30 12:25:51.910336 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.890781 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-token\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.910591 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.891867 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcgqj\" (UniqueName: \"kubernetes.io/projected/f1e658f2-c95e-40fc-9602-ac8271892d05-kube-api-access-bcgqj\") pod \"e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:51.975287 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:51.974505 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:52.315153 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:52.315061 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp"] Oct 30 12:25:52.321887 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:25:52.321857 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e658f2_c95e_40fc_9602_ac8271892d05.slice/crio-793c542ee6d76135645fa678685f7f8853236536097b6026803cb6037e93631f WatchSource:0}: Error finding container 793c542ee6d76135645fa678685f7f8853236536097b6026803cb6037e93631f: Status 404 returned error can't find the container with id 793c542ee6d76135645fa678685f7f8853236536097b6026803cb6037e93631f Oct 30 12:25:52.325787 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:52.325710 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:25:52.326032 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:52.325819 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:25:52.326032 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:52.325867 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:25:52.772424 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:52.772388 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" event={"ID":"f1e658f2-c95e-40fc-9602-ac8271892d05","Type":"ContainerStarted","Data":"a6b8bd67a912193cde8efa8f34e6233230f257fdc1b0c337753c74af4bdb575c"} Oct 30 12:25:52.772913 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:52.772452 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" event={"ID":"f1e658f2-c95e-40fc-9602-ac8271892d05","Type":"ContainerStarted","Data":"793c542ee6d76135645fa678685f7f8853236536097b6026803cb6037e93631f"} Oct 30 12:25:52.786654 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:52.786623 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:52.828452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:52.828312 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" podStartSLOduration=1.828292561 podStartE2EDuration="1.828292561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:52.823911717 +0000 UTC m=+1905.933754601" watchObservedRunningTime="2026-10-30 12:25:52.828292561 +0000 UTC m=+1905.938135443" Oct 30 12:25:52.975311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:52.975281 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:53.981394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:53.981367 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:54.307075 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:54.306995 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:54.419121 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:54.419088 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:54.549809 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:54.549776 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:54.721445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:54.721378 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:54.788068 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:54.788034 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:54.789678 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:54.789637 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:25:55.676123 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:55.676085 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:55.815468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:55.815441 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:57.618831 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:57.618796 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:57.790956 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:57.790915 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:57.995552 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:57.995520 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:25:58.154921 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:58.154889 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z"] Oct 30 12:25:58.162751 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:58.162725 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z" Oct 30 12:25:58.176277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:58.176209 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z"] Oct 30 12:25:58.186288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:58.179963 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-266486-dockercfg-ch5bx\"" Oct 30 12:25:58.186288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:58.180088 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-266486\"" Oct 30 12:25:58.223274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:58.212322 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khg29\" (UniqueName: \"kubernetes.io/projected/e4d03a58-07e0-429f-bb8b-b3a9a0f09164-kube-api-access-khg29\") pod \"payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z\" (UID: \"e4d03a58-07e0-429f-bb8b-b3a9a0f09164\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z" Oct 30 12:25:58.223274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:58.212627 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d03a58-07e0-429f-bb8b-b3a9a0f09164-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z\" (UID: \"e4d03a58-07e0-429f-bb8b-b3a9a0f09164\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z" Oct 30 12:25:58.316756 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:58.316660 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d03a58-07e0-429f-bb8b-b3a9a0f09164-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z\" (UID: \"e4d03a58-07e0-429f-bb8b-b3a9a0f09164\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z" Oct 30 12:25:58.316922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:58.316753 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khg29\" (UniqueName: \"kubernetes.io/projected/e4d03a58-07e0-429f-bb8b-b3a9a0f09164-kube-api-access-khg29\") pod \"payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z\" (UID: \"e4d03a58-07e0-429f-bb8b-b3a9a0f09164\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z" Oct 30 12:25:58.317591 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:58.317564 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d03a58-07e0-429f-bb8b-b3a9a0f09164-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z\" (UID: \"e4d03a58-07e0-429f-bb8b-b3a9a0f09164\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z" Oct 30 12:25:58.343347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:58.343320 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khg29\" (UniqueName: \"kubernetes.io/projected/e4d03a58-07e0-429f-bb8b-b3a9a0f09164-kube-api-access-khg29\") pod \"payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z\" (UID: \"e4d03a58-07e0-429f-bb8b-b3a9a0f09164\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z" Oct 30 12:25:58.525693 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:58.525653 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z" Oct 30 12:25:58.832437 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:58.832415 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z"] Oct 30 12:25:58.832963 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:25:58.832941 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d03a58_07e0_429f_bb8b_b3a9a0f09164.slice/crio-02110652239a3452ba9a0f264b85761f82897efa247406f5f33272e5fca6fd61 WatchSource:0}: Error finding container 02110652239a3452ba9a0f264b85761f82897efa247406f5f33272e5fca6fd61: Status 404 returned error can't find the container with id 02110652239a3452ba9a0f264b85761f82897efa247406f5f33272e5fca6fd61 Oct 30 12:25:59.823582 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:59.823531 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z" event={"ID":"e4d03a58-07e0-429f-bb8b-b3a9a0f09164","Type":"ContainerStarted","Data":"8b06d1233d4983874efc63e50f445758cb616634a2564727558f7bd1b832bc8f"} Oct 30 12:25:59.823790 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:59.823609 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z" Oct 30 12:25:59.823790 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:59.823633 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z" event={"ID":"e4d03a58-07e0-429f-bb8b-b3a9a0f09164","Type":"ContainerStarted","Data":"02110652239a3452ba9a0f264b85761f82897efa247406f5f33272e5fca6fd61"} Oct 30 12:25:59.863577 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:25:59.863524 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z" podStartSLOduration=1.863511744 podStartE2EDuration="1.863511744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:59.862983329 +0000 UTC m=+1912.972826211" watchObservedRunningTime="2026-10-30 12:25:59.863511744 +0000 UTC m=+1912.973354625" Oct 30 12:26:00.047698 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:00.047663 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:01.802929 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:01.802894 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:02.776003 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:02.775977 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv" Oct 30 12:26:02.776187 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:02.776106 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24" Oct 30 12:26:03.073763 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:03.073670 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:03.248607 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:03.248574 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:03.863674 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:03.863642 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf"] Oct 30 12:26:03.863879 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:03.863850 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" podUID="3f079200-ed6a-4ed5-b378-9f12a141e464" containerName="istio-proxy" containerID="cri-o://930ce6b1eead00224d7604c304066552d5df3ac5e7a0c5f20624d96aabfb3340" gracePeriod=30 Oct 30 12:26:04.407629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:04.407593 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:05.682749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:05.682712 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:05.847141 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:05.847099 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:05.875215 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:26:05.875180 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:26:06.091687 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:06.091597 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk"] Oct 30 12:26:06.098313 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:06.097623 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" Oct 30 12:26:06.109020 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:06.108993 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk"] Oct 30 12:26:06.121117 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:06.120425 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7e2db2\"" Oct 30 12:26:06.121117 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:06.120996 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7e2db2-dockercfg-sbxh9\"" Oct 30 12:26:06.174285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:06.174229 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjcnn\" (UniqueName: \"kubernetes.io/projected/c0ee5a0e-d8d9-4823-9ebd-512e246c744b-kube-api-access-gjcnn\") pod \"payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk\" (UID: \"c0ee5a0e-d8d9-4823-9ebd-512e246c744b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" Oct 30 12:26:06.174457 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:06.174354 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ee5a0e-d8d9-4823-9ebd-512e246c744b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk\" (UID: \"c0ee5a0e-d8d9-4823-9ebd-512e246c744b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" Oct 30 12:26:06.275692 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:06.275653 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ee5a0e-d8d9-4823-9ebd-512e246c744b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk\" (UID: \"c0ee5a0e-d8d9-4823-9ebd-512e246c744b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" Oct 30 12:26:06.276222 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:06.276195 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjcnn\" (UniqueName: \"kubernetes.io/projected/c0ee5a0e-d8d9-4823-9ebd-512e246c744b-kube-api-access-gjcnn\") pod \"payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk\" (UID: \"c0ee5a0e-d8d9-4823-9ebd-512e246c744b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" Oct 30 12:26:06.276748 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:06.276709 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ee5a0e-d8d9-4823-9ebd-512e246c744b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk\" (UID: \"c0ee5a0e-d8d9-4823-9ebd-512e246c744b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" Oct 30 12:26:06.299819 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:06.299787 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjcnn\" (UniqueName: \"kubernetes.io/projected/c0ee5a0e-d8d9-4823-9ebd-512e246c744b-kube-api-access-gjcnn\") pod \"payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk\" (UID: \"c0ee5a0e-d8d9-4823-9ebd-512e246c744b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" Oct 30 12:26:06.438009 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:06.437959 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" Oct 30 12:26:06.744745 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:06.744704 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk"] Oct 30 12:26:06.745797 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:26:06.745770 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ee5a0e_d8d9_4823_9ebd_512e246c744b.slice/crio-a2e55bd291f7528d30d8e494bd472332446fc34c369d2fa97895f5e1a8fb4836 WatchSource:0}: Error finding container a2e55bd291f7528d30d8e494bd472332446fc34c369d2fa97895f5e1a8fb4836: Status 404 returned error can't find the container with id a2e55bd291f7528d30d8e494bd472332446fc34c369d2fa97895f5e1a8fb4836 Oct 30 12:26:06.748114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:06.748094 4413 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:26:06.885530 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:06.885492 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" event={"ID":"c0ee5a0e-d8d9-4823-9ebd-512e246c744b","Type":"ContainerStarted","Data":"a2e55bd291f7528d30d8e494bd472332446fc34c369d2fa97895f5e1a8fb4836"} Oct 30 12:26:06.952582 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:06.952541 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:07.894574 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:07.894540 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" event={"ID":"c0ee5a0e-d8d9-4823-9ebd-512e246c744b","Type":"ContainerStarted","Data":"d7763c84e920fbf347e2233c04eceda565bc2fd883a184788f568ead10df20d5"} Oct 30 12:26:07.895033 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:07.894713 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" Oct 30 12:26:07.935855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:07.935793 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" podStartSLOduration=0.876983723 podStartE2EDuration="1.935778379s" totalImagesPullingTime="1.058794656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:07.934323362 +0000 UTC m=+1921.044166266" watchObservedRunningTime="2026-10-30 12:26:07.935778379 +0000 UTC m=+1921.045621292" Oct 30 12:26:08.807851 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:08.807812 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:09.162353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.162329 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:26:09.229093 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.229060 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-workload-socket\") pod \"3f079200-ed6a-4ed5-b378-9f12a141e464\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " Oct 30 12:26:09.229286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.229133 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-data\") pod \"3f079200-ed6a-4ed5-b378-9f12a141e464\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " Oct 30 12:26:09.229286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.229228 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f079200-ed6a-4ed5-b378-9f12a141e464-kube-api-access-2cprf\" (UniqueName: \"kubernetes.io/projected/3f079200-ed6a-4ed5-b378-9f12a141e464-kube-api-access-2cprf\") pod \"3f079200-ed6a-4ed5-b378-9f12a141e464\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " Oct 30 12:26:09.229442 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.229347 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-token\" (UniqueName: \"kubernetes.io/projected/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-token\") pod \"3f079200-ed6a-4ed5-b378-9f12a141e464\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " Oct 30 12:26:09.229442 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.229387 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-workload-certs\") pod \"3f079200-ed6a-4ed5-b378-9f12a141e464\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " Oct 30 12:26:09.229551 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.229541 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-envoy\") pod \"3f079200-ed6a-4ed5-b378-9f12a141e464\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " Oct 30 12:26:09.229608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.229589 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-podinfo\") pod \"3f079200-ed6a-4ed5-b378-9f12a141e464\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " Oct 30 12:26:09.229665 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.229611 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-workload-socket" pod "3f079200-ed6a-4ed5-b378-9f12a141e464" (UID: "3f079200-ed6a-4ed5-b378-9f12a141e464"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:09.229665 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.229619 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-credential-socket\") pod \"3f079200-ed6a-4ed5-b378-9f12a141e464\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " Oct 30 12:26:09.229665 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.229650 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f079200-ed6a-4ed5-b378-9f12a141e464-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f079200-ed6a-4ed5-b378-9f12a141e464-istiod-ca-cert\") pod \"3f079200-ed6a-4ed5-b378-9f12a141e464\" (UID: \"3f079200-ed6a-4ed5-b378-9f12a141e464\") " Oct 30 12:26:09.230044 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.229996 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-data" pod "3f079200-ed6a-4ed5-b378-9f12a141e464" (UID: "3f079200-ed6a-4ed5-b378-9f12a141e464"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:09.230436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.230335 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.230825 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.230801 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f079200-ed6a-4ed5-b378-9f12a141e464-istiod-ca-cert" pod "3f079200-ed6a-4ed5-b378-9f12a141e464" (UID: "3f079200-ed6a-4ed5-b378-9f12a141e464"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:26:09.230943 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.230813 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-credential-socket" pod "3f079200-ed6a-4ed5-b378-9f12a141e464" (UID: "3f079200-ed6a-4ed5-b378-9f12a141e464"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:09.231484 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.231455 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-workload-certs" pod "3f079200-ed6a-4ed5-b378-9f12a141e464" (UID: "3f079200-ed6a-4ed5-b378-9f12a141e464"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:09.232683 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.232659 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-podinfo" pod "3f079200-ed6a-4ed5-b378-9f12a141e464" (UID: "3f079200-ed6a-4ed5-b378-9f12a141e464"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:26:09.233650 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.233620 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-envoy" pod "3f079200-ed6a-4ed5-b378-9f12a141e464" (UID: "3f079200-ed6a-4ed5-b378-9f12a141e464"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:09.234175 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.234155 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-token" pod "3f079200-ed6a-4ed5-b378-9f12a141e464" (UID: "3f079200-ed6a-4ed5-b378-9f12a141e464"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:26:09.236860 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.236830 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f079200-ed6a-4ed5-b378-9f12a141e464-kube-api-access-2cprf" pod "3f079200-ed6a-4ed5-b378-9f12a141e464" (UID: "3f079200-ed6a-4ed5-b378-9f12a141e464"). InnerVolumeSpecName "kube-api-access-2cprf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:26:09.331905 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.331877 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.331905 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.331906 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.332087 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.331919 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.332087 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.331933 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f079200-ed6a-4ed5-b378-9f12a141e464-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.332087 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.331946 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.332087 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.331955 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cprf\" (UniqueName: \"kubernetes.io/projected/3f079200-ed6a-4ed5-b378-9f12a141e464-kube-api-access-2cprf\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.332087 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.331963 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f079200-ed6a-4ed5-b378-9f12a141e464-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.332087 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.331971 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f079200-ed6a-4ed5-b378-9f12a141e464-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:09.913413 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.913376 4413 generic.go:356] "Generic (PLEG): container finished" podID="3f079200-ed6a-4ed5-b378-9f12a141e464" containerID="930ce6b1eead00224d7604c304066552d5df3ac5e7a0c5f20624d96aabfb3340" exitCode=0 Oct 30 12:26:09.913616 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.913431 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" Oct 30 12:26:09.913616 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.913435 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" event={"ID":"3f079200-ed6a-4ed5-b378-9f12a141e464","Type":"ContainerDied","Data":"930ce6b1eead00224d7604c304066552d5df3ac5e7a0c5f20624d96aabfb3340"} Oct 30 12:26:09.913616 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.913491 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf" event={"ID":"3f079200-ed6a-4ed5-b378-9f12a141e464","Type":"ContainerDied","Data":"80874ea33f7489c714f825db9868275aed1179de04b1c2b15b43f21d4014d12a"} Oct 30 12:26:09.913616 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.913520 4413 scope.go:122] "RemoveContainer" containerID="930ce6b1eead00224d7604c304066552d5df3ac5e7a0c5f20624d96aabfb3340" Oct 30 12:26:09.923961 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.923942 4413 scope.go:122] "RemoveContainer" containerID="930ce6b1eead00224d7604c304066552d5df3ac5e7a0c5f20624d96aabfb3340" Oct 30 12:26:09.924262 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:26:09.924232 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930ce6b1eead00224d7604c304066552d5df3ac5e7a0c5f20624d96aabfb3340\": container with ID starting with 930ce6b1eead00224d7604c304066552d5df3ac5e7a0c5f20624d96aabfb3340 not found: ID does not exist" containerID="930ce6b1eead00224d7604c304066552d5df3ac5e7a0c5f20624d96aabfb3340" Oct 30 12:26:09.924329 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.924273 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930ce6b1eead00224d7604c304066552d5df3ac5e7a0c5f20624d96aabfb3340"} err="failed to get container status \"930ce6b1eead00224d7604c304066552d5df3ac5e7a0c5f20624d96aabfb3340\": rpc error: code = NotFound desc = could not find container \"930ce6b1eead00224d7604c304066552d5df3ac5e7a0c5f20624d96aabfb3340\": container with ID starting with 930ce6b1eead00224d7604c304066552d5df3ac5e7a0c5f20624d96aabfb3340 not found: ID does not exist" Oct 30 12:26:09.964491 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.964465 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf"] Oct 30 12:26:09.976187 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:09.976157 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-4vzpf"] Oct 30 12:26:10.575503 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:10.575470 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:10.832729 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:10.832654 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z" Oct 30 12:26:11.021088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:11.021052 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f079200-ed6a-4ed5-b378-9f12a141e464" path="/var/lib/kubelet/pods/3f079200-ed6a-4ed5-b378-9f12a141e464/volumes" Oct 30 12:26:12.160230 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.160197 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv"] Oct 30 12:26:12.160853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.160839 4413 cpu_manager.go:414] "containerMap: removing container" podUID="3f079200-ed6a-4ed5-b378-9f12a141e464" containerName="istio-proxy" Oct 30 12:26:12.160897 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.160854 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f079200-ed6a-4ed5-b378-9f12a141e464" containerName="istio-proxy" Oct 30 12:26:12.160966 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.160957 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f079200-ed6a-4ed5-b378-9f12a141e464" containerName="istio-proxy" Oct 30 12:26:12.165907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.165888 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" Oct 30 12:26:12.186374 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.186348 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv"] Oct 30 12:26:12.192358 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.192336 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-431bf6-serving-cert\"" Oct 30 12:26:12.255701 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.255669 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:12.270210 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.270184 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw7dn\" (UniqueName: \"kubernetes.io/projected/02adbaa5-0ff4-4471-bf4a-d81d9207a485-kube-api-access-cw7dn\") pod \"maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv\" (UID: \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" Oct 30 12:26:12.270346 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.270328 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02adbaa5-0ff4-4471-bf4a-d81d9207a485-metrics-certs\") pod \"maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv\" (UID: \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" Oct 30 12:26:12.270393 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.270371 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02adbaa5-0ff4-4471-bf4a-d81d9207a485-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv\" (UID: \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" Oct 30 12:26:12.270534 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.270520 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02adbaa5-0ff4-4471-bf4a-d81d9207a485-maas-api-tls\") pod \"maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv\" (UID: \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" Oct 30 12:26:12.368565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.368521 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:12.372026 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.371988 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02adbaa5-0ff4-4471-bf4a-d81d9207a485-metrics-certs\") pod \"maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv\" (UID: \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" Oct 30 12:26:12.372166 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.372043 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02adbaa5-0ff4-4471-bf4a-d81d9207a485-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv\" (UID: \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" Oct 30 12:26:12.372212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.372172 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02adbaa5-0ff4-4471-bf4a-d81d9207a485-maas-api-tls\") pod \"maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv\" (UID: \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" Oct 30 12:26:12.372330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.372301 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw7dn\" (UniqueName: \"kubernetes.io/projected/02adbaa5-0ff4-4471-bf4a-d81d9207a485-kube-api-access-cw7dn\") pod \"maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv\" (UID: \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" Oct 30 12:26:12.372805 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.372735 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02adbaa5-0ff4-4471-bf4a-d81d9207a485-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv\" (UID: \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" Oct 30 12:26:12.374683 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.374658 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02adbaa5-0ff4-4471-bf4a-d81d9207a485-metrics-certs\") pod \"maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv\" (UID: \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" Oct 30 12:26:12.374785 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.374707 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02adbaa5-0ff4-4471-bf4a-d81d9207a485-maas-api-tls\") pod \"maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv\" (UID: \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" Oct 30 12:26:12.395211 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.395186 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw7dn\" (UniqueName: \"kubernetes.io/projected/02adbaa5-0ff4-4471-bf4a-d81d9207a485-kube-api-access-cw7dn\") pod \"maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv\" (UID: \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" Oct 30 12:26:12.479872 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.479776 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" Oct 30 12:26:12.482734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.482704 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:12.793801 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.793776 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv"] Oct 30 12:26:12.794674 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:26:12.794653 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02adbaa5_0ff4_4471_bf4a_d81d9207a485.slice/crio-c9683621c9493833fd3d9cecdbb1cf75114be17d7d4a9f9a943da2d09b9ed7f8 WatchSource:0}: Error finding container c9683621c9493833fd3d9cecdbb1cf75114be17d7d4a9f9a943da2d09b9ed7f8: Status 404 returned error can't find the container with id c9683621c9493833fd3d9cecdbb1cf75114be17d7d4a9f9a943da2d09b9ed7f8 Oct 30 12:26:12.935830 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:12.935796 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" event={"ID":"02adbaa5-0ff4-4471-bf4a-d81d9207a485","Type":"ContainerStarted","Data":"c9683621c9493833fd3d9cecdbb1cf75114be17d7d4a9f9a943da2d09b9ed7f8"} Oct 30 12:26:13.944156 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:13.944120 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" event={"ID":"02adbaa5-0ff4-4471-bf4a-d81d9207a485","Type":"ContainerStarted","Data":"fee6bf37af89925ea5fb614f8f4007670224280a9f2ff7d2c14518540a76b376"} Oct 30 12:26:13.944590 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:13.944277 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" Oct 30 12:26:13.985965 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:13.985908 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" podStartSLOduration=1.618453917 podStartE2EDuration="1.985891126s" totalImagesPullingTime="367.437209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:13.984195318 +0000 UTC m=+1927.094038217" watchObservedRunningTime="2026-10-30 12:26:13.985891126 +0000 UTC m=+1927.095734014" Oct 30 12:26:14.239954 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:14.239833 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:15.477063 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:15.477029 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:15.599820 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:15.599789 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:16.578383 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:16.578336 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:18.244460 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:18.244429 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:18.907445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:18.907418 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" Oct 30 12:26:19.807325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:19.807289 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq"] Oct 30 12:26:19.811786 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:19.811764 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" Oct 30 12:26:19.827462 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:19.827057 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-266486-serving-cert\"" Oct 30 12:26:19.843187 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:19.843146 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq"] Oct 30 12:26:19.877389 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:19.877359 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/943846d2-b5db-4c25-94e8-e364fcb506f9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq\" (UID: \"943846d2-b5db-4c25-94e8-e364fcb506f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" Oct 30 12:26:19.877560 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:19.877529 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2v89\" (UniqueName: \"kubernetes.io/projected/943846d2-b5db-4c25-94e8-e364fcb506f9-kube-api-access-v2v89\") pod \"maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq\" (UID: \"943846d2-b5db-4c25-94e8-e364fcb506f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" Oct 30 12:26:19.877610 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:19.877571 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/943846d2-b5db-4c25-94e8-e364fcb506f9-metrics-certs\") pod \"maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq\" (UID: \"943846d2-b5db-4c25-94e8-e364fcb506f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" Oct 30 12:26:19.877730 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:19.877701 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/943846d2-b5db-4c25-94e8-e364fcb506f9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq\" (UID: \"943846d2-b5db-4c25-94e8-e364fcb506f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" Oct 30 12:26:19.897320 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:19.897292 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:19.955606 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:19.955583 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" Oct 30 12:26:19.978646 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:19.978620 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/943846d2-b5db-4c25-94e8-e364fcb506f9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq\" (UID: \"943846d2-b5db-4c25-94e8-e364fcb506f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" Oct 30 12:26:19.978775 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:19.978724 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2v89\" (UniqueName: \"kubernetes.io/projected/943846d2-b5db-4c25-94e8-e364fcb506f9-kube-api-access-v2v89\") pod \"maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq\" (UID: \"943846d2-b5db-4c25-94e8-e364fcb506f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" Oct 30 12:26:19.978775 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:19.978765 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/943846d2-b5db-4c25-94e8-e364fcb506f9-metrics-certs\") pod \"maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq\" (UID: \"943846d2-b5db-4c25-94e8-e364fcb506f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" Oct 30 12:26:19.978902 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:19.978813 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/943846d2-b5db-4c25-94e8-e364fcb506f9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq\" (UID: \"943846d2-b5db-4c25-94e8-e364fcb506f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" Oct 30 12:26:19.979541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:19.979519 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/943846d2-b5db-4c25-94e8-e364fcb506f9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq\" (UID: \"943846d2-b5db-4c25-94e8-e364fcb506f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" Oct 30 12:26:19.981480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:19.981461 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/943846d2-b5db-4c25-94e8-e364fcb506f9-metrics-certs\") pod \"maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq\" (UID: \"943846d2-b5db-4c25-94e8-e364fcb506f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" Oct 30 12:26:19.981683 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:19.981666 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/943846d2-b5db-4c25-94e8-e364fcb506f9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq\" (UID: \"943846d2-b5db-4c25-94e8-e364fcb506f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" Oct 30 12:26:20.008365 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:20.008335 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2v89\" (UniqueName: \"kubernetes.io/projected/943846d2-b5db-4c25-94e8-e364fcb506f9-kube-api-access-v2v89\") pod \"maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq\" (UID: \"943846d2-b5db-4c25-94e8-e364fcb506f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" Oct 30 12:26:20.024854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:20.024819 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:20.127570 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:20.127546 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" Oct 30 12:26:20.137706 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:20.137674 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:20.440305 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:20.440272 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq"] Oct 30 12:26:20.440932 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:26:20.440908 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943846d2_b5db_4c25_94e8_e364fcb506f9.slice/crio-a6902c3b8daa98f63947226f97d21be191f2d24eb15a4b89853a108942f15e58 WatchSource:0}: Error finding container a6902c3b8daa98f63947226f97d21be191f2d24eb15a4b89853a108942f15e58: Status 404 returned error can't find the container with id a6902c3b8daa98f63947226f97d21be191f2d24eb15a4b89853a108942f15e58 Oct 30 12:26:21.005040 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:21.005008 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" event={"ID":"943846d2-b5db-4c25-94e8-e364fcb506f9","Type":"ContainerStarted","Data":"a6902c3b8daa98f63947226f97d21be191f2d24eb15a4b89853a108942f15e58"} Oct 30 12:26:21.278195 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:21.278101 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6"] Oct 30 12:26:21.278396 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:21.278370 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" podUID="c842dc5f-e6cc-4133-bd37-6c95f466adca" containerName="maas-api" containerID="cri-o://d07bebaef464ef602373c0ffa4fc7568b2af74e084d74e9daa34e64f7de748f4" gracePeriod=30 Oct 30 12:26:21.344263 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:21.344221 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:21.627048 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:21.627021 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:26:21.705272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:21.705002 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c842dc5f-e6cc-4133-bd37-6c95f466adca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c842dc5f-e6cc-4133-bd37-6c95f466adca-maas-api-tls\") pod \"c842dc5f-e6cc-4133-bd37-6c95f466adca\" (UID: \"c842dc5f-e6cc-4133-bd37-6c95f466adca\") " Oct 30 12:26:21.705272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:21.705179 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c842dc5f-e6cc-4133-bd37-6c95f466adca-kube-api-access-smtd9\" (UniqueName: \"kubernetes.io/projected/c842dc5f-e6cc-4133-bd37-6c95f466adca-kube-api-access-smtd9\") pod \"c842dc5f-e6cc-4133-bd37-6c95f466adca\" (UID: \"c842dc5f-e6cc-4133-bd37-6c95f466adca\") " Oct 30 12:26:21.705499 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:21.705424 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c842dc5f-e6cc-4133-bd37-6c95f466adca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c842dc5f-e6cc-4133-bd37-6c95f466adca-openshift-service-ca\") pod \"c842dc5f-e6cc-4133-bd37-6c95f466adca\" (UID: \"c842dc5f-e6cc-4133-bd37-6c95f466adca\") " Oct 30 12:26:21.705499 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:21.705463 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c842dc5f-e6cc-4133-bd37-6c95f466adca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c842dc5f-e6cc-4133-bd37-6c95f466adca-metrics-certs\") pod \"c842dc5f-e6cc-4133-bd37-6c95f466adca\" (UID: \"c842dc5f-e6cc-4133-bd37-6c95f466adca\") " Oct 30 12:26:21.706784 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:21.705832 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c842dc5f-e6cc-4133-bd37-6c95f466adca-openshift-service-ca" pod "c842dc5f-e6cc-4133-bd37-6c95f466adca" (UID: "c842dc5f-e6cc-4133-bd37-6c95f466adca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:26:21.706784 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:21.706362 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c842dc5f-e6cc-4133-bd37-6c95f466adca-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:21.711067 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:21.710856 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c842dc5f-e6cc-4133-bd37-6c95f466adca-metrics-certs" pod "c842dc5f-e6cc-4133-bd37-6c95f466adca" (UID: "c842dc5f-e6cc-4133-bd37-6c95f466adca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:26:21.711067 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:21.710954 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c842dc5f-e6cc-4133-bd37-6c95f466adca-maas-api-tls" pod "c842dc5f-e6cc-4133-bd37-6c95f466adca" (UID: "c842dc5f-e6cc-4133-bd37-6c95f466adca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:26:21.712830 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:21.712801 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c842dc5f-e6cc-4133-bd37-6c95f466adca-kube-api-access-smtd9" pod "c842dc5f-e6cc-4133-bd37-6c95f466adca" (UID: "c842dc5f-e6cc-4133-bd37-6c95f466adca"). InnerVolumeSpecName "kube-api-access-smtd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:26:21.808924 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:21.808564 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c842dc5f-e6cc-4133-bd37-6c95f466adca-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:21.808924 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:21.808597 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c842dc5f-e6cc-4133-bd37-6c95f466adca-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:21.808924 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:21.808609 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smtd9\" (UniqueName: \"kubernetes.io/projected/c842dc5f-e6cc-4133-bd37-6c95f466adca-kube-api-access-smtd9\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:21.949839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:21.949807 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:22.014891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:22.014855 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" event={"ID":"943846d2-b5db-4c25-94e8-e364fcb506f9","Type":"ContainerStarted","Data":"2fe34869c8d88a53635474a13eca22edcddeec7123719891c8d76e8a3ec9d611"} Oct 30 12:26:22.015399 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:22.014918 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" Oct 30 12:26:22.016189 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:22.016163 4413 generic.go:356] "Generic (PLEG): container finished" podID="c842dc5f-e6cc-4133-bd37-6c95f466adca" containerID="d07bebaef464ef602373c0ffa4fc7568b2af74e084d74e9daa34e64f7de748f4" exitCode=0 Oct 30 12:26:22.016334 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:22.016201 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" Oct 30 12:26:22.016334 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:22.016277 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" event={"ID":"c842dc5f-e6cc-4133-bd37-6c95f466adca","Type":"ContainerDied","Data":"d07bebaef464ef602373c0ffa4fc7568b2af74e084d74e9daa34e64f7de748f4"} Oct 30 12:26:22.016334 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:22.016322 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6" event={"ID":"c842dc5f-e6cc-4133-bd37-6c95f466adca","Type":"ContainerDied","Data":"b7e7af5481fc69b3ad4b71a9c5649f608fc6a7dac642ad4786be44d4f7ab267e"} Oct 30 12:26:22.016498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:22.016350 4413 scope.go:122] "RemoveContainer" containerID="d07bebaef464ef602373c0ffa4fc7568b2af74e084d74e9daa34e64f7de748f4" Oct 30 12:26:22.029198 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:22.029174 4413 scope.go:122] "RemoveContainer" containerID="d07bebaef464ef602373c0ffa4fc7568b2af74e084d74e9daa34e64f7de748f4" Oct 30 12:26:22.029518 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:26:22.029498 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d07bebaef464ef602373c0ffa4fc7568b2af74e084d74e9daa34e64f7de748f4\": container with ID starting with d07bebaef464ef602373c0ffa4fc7568b2af74e084d74e9daa34e64f7de748f4 not found: ID does not exist" containerID="d07bebaef464ef602373c0ffa4fc7568b2af74e084d74e9daa34e64f7de748f4" Oct 30 12:26:22.029586 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:22.029525 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07bebaef464ef602373c0ffa4fc7568b2af74e084d74e9daa34e64f7de748f4"} err="failed to get container status \"d07bebaef464ef602373c0ffa4fc7568b2af74e084d74e9daa34e64f7de748f4\": rpc error: code = NotFound desc = could not find container \"d07bebaef464ef602373c0ffa4fc7568b2af74e084d74e9daa34e64f7de748f4\": container with ID starting with d07bebaef464ef602373c0ffa4fc7568b2af74e084d74e9daa34e64f7de748f4 not found: ID does not exist" Oct 30 12:26:22.057519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:22.057477 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" podStartSLOduration=2.554032641 podStartE2EDuration="3.057464882s" totalImagesPullingTime="503.432241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:22.055243082 +0000 UTC m=+1935.165085981" watchObservedRunningTime="2026-10-30 12:26:22.057464882 +0000 UTC m=+1935.167307763" Oct 30 12:26:22.067596 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:22.067570 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:22.091570 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:22.091522 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6"] Oct 30 12:26:22.098599 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:22.098554 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-922643-69c598bf54-2cgg6"] Oct 30 12:26:22.185322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:22.185290 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:22.634960 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:22.634927 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:23.020518 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:23.020446 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c842dc5f-e6cc-4133-bd37-6c95f466adca" path="/var/lib/kubelet/pods/c842dc5f-e6cc-4133-bd37-6c95f466adca/volumes" Oct 30 12:26:24.326067 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:24.326034 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:25.300583 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:25.300553 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:25.799468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:25.799431 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt"] Oct 30 12:26:25.800476 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:25.800452 4413 cpu_manager.go:414] "containerMap: removing container" podUID="c842dc5f-e6cc-4133-bd37-6c95f466adca" containerName="maas-api" Oct 30 12:26:25.800596 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:25.800478 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c842dc5f-e6cc-4133-bd37-6c95f466adca" containerName="maas-api" Oct 30 12:26:25.800647 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:25.800635 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="c842dc5f-e6cc-4133-bd37-6c95f466adca" containerName="maas-api" Oct 30 12:26:25.806502 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:25.806480 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:25.823156 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:25.821345 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-096834-openshift-default-dockercfg-bbrhx\"" Oct 30 12:26:25.845984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:25.845143 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt"] Oct 30 12:26:25.952713 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:25.952676 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:25.973818 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:25.973752 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-credential-socket\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:25.974038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:25.973838 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-workload-certs\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:25.974038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:25.973892 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-envoy\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:25.974038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:25.973955 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-workload-socket\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:25.974038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:25.974025 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-data\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:25.974313 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:25.974201 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-token\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:25.974313 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:25.974232 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8r8n\" (UniqueName: \"kubernetes.io/projected/aa0c1522-725d-43f2-abd8-c37053a1f70c-kube-api-access-k8r8n\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:25.974657 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:25.974630 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-podinfo\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:25.974817 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:25.974798 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa0c1522-725d-43f2-abd8-c37053a1f70c-istiod-ca-cert\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.076956 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.076859 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-workload-socket\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.076956 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.076934 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-data\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.077176 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.077062 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-token\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.077176 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.077091 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8r8n\" (UniqueName: \"kubernetes.io/projected/aa0c1522-725d-43f2-abd8-c37053a1f70c-kube-api-access-k8r8n\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.078376 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.078344 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-workload-socket\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.078658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.078635 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-data\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.079071 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.077265 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-podinfo\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.079156 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.079120 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa0c1522-725d-43f2-abd8-c37053a1f70c-istiod-ca-cert\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.079218 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.079184 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-credential-socket\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.079310 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.079225 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-workload-certs\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.079310 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.079276 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-envoy\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.079933 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.079909 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-credential-socket\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.081501 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.080301 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-workload-certs\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.081731 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.080366 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa0c1522-725d-43f2-abd8-c37053a1f70c-istiod-ca-cert\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.118599 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.103375 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-podinfo\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.118599 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.103747 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-envoy\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.118599 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.109776 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-token\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.119478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.119423 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8r8n\" (UniqueName: \"kubernetes.io/projected/aa0c1522-725d-43f2-abd8-c37053a1f70c-kube-api-access-k8r8n\") pod \"e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.128383 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.127805 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:26.442790 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.442755 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:26.527241 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.527208 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt"] Oct 30 12:26:26.530543 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:26:26.530514 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0c1522_725d_43f2_abd8_c37053a1f70c.slice/crio-43fd3fefc3d9ae743bda242b478a5a06926863e907e943255a474b5c8dad7fba WatchSource:0}: Error finding container 43fd3fefc3d9ae743bda242b478a5a06926863e907e943255a474b5c8dad7fba: Status 404 returned error can't find the container with id 43fd3fefc3d9ae743bda242b478a5a06926863e907e943255a474b5c8dad7fba Oct 30 12:26:26.533232 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.533190 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:26:26.533370 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.533293 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:26:26.533370 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.533343 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:26:26.746760 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:26.746580 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:27.049854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:27.049781 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" event={"ID":"aa0c1522-725d-43f2-abd8-c37053a1f70c","Type":"ContainerStarted","Data":"de2579e32ebcd73db1abfd137677870b35b605be3cce623e19a42358a1cdabdf"} Oct 30 12:26:27.049854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:27.049831 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" event={"ID":"aa0c1522-725d-43f2-abd8-c37053a1f70c","Type":"ContainerStarted","Data":"43fd3fefc3d9ae743bda242b478a5a06926863e907e943255a474b5c8dad7fba"} Oct 30 12:26:27.112001 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:27.105393 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" podStartSLOduration=2.105373078 podStartE2EDuration="2.105373078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:27.102778604 +0000 UTC m=+1940.212621485" watchObservedRunningTime="2026-10-30 12:26:27.105373078 +0000 UTC m=+1940.215215966" Oct 30 12:26:27.128705 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:27.128675 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:27.134885 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:27.134862 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:28.029231 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:28.029199 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" Oct 30 12:26:28.056321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:28.056284 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:28.057101 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:28.057083 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:26:28.416812 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:28.416765 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:29.059813 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:29.059779 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:29.172530 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:29.172501 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:29.427275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:29.427219 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz"] Oct 30 12:26:29.427529 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:29.427498 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" podUID="67aa5abb-dd1e-4ac9-888e-bb958fddffa9" containerName="istio-proxy" containerID="cri-o://762ee8d14c218d84d114435177787d40ac93425aecb31ee70027fee69949bf42" gracePeriod=30 Oct 30 12:26:29.448307 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:29.448233 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:29.575765 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:29.575729 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:29.704212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:29.704109 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:30.915290 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:30.910862 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:31.315932 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:31.315828 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:31.553484 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:31.553453 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:31.673271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:31.672663 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:32.022503 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.021950 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw"] Oct 30 12:26:32.030668 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.030641 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.033953 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.033924 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-envoy\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.034103 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.034020 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-data\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.034187 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.034124 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-token\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.034187 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.034159 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-credential-socket\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.034342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.034191 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istiod-ca-cert\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.034342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.034222 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-workload-certs\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.034342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.034266 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-workload-socket\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.034342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.034332 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qq9h\" (UniqueName: \"kubernetes.io/projected/2cb74db0-df6e-471c-bce4-7de34b2a33e3-kube-api-access-4qq9h\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.034558 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.034448 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-podinfo\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.060426 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.060392 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw"] Oct 30 12:26:32.077924 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.076460 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5c57af-openshift-default-dockercfg-vkw5v\"" Oct 30 12:26:32.118614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.118582 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:32.135720 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.135660 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-podinfo\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.135909 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.135744 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-envoy\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.135909 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.135844 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-data\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.136151 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.136126 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-token\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.136291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.136182 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-credential-socket\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.136291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.136216 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istiod-ca-cert\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.136545 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.136295 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-workload-certs\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.136545 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.136325 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-workload-socket\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.136545 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.136414 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qq9h\" (UniqueName: \"kubernetes.io/projected/2cb74db0-df6e-471c-bce4-7de34b2a33e3-kube-api-access-4qq9h\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.137838 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.137439 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-workload-certs\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.137838 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.137461 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istiod-ca-cert\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.137838 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.137750 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-workload-socket\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.137838 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.137777 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-credential-socket\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.138067 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.137848 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-data\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.138783 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.138764 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-envoy\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.139794 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.139778 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-podinfo\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.171441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.171411 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-token\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.173740 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.173709 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qq9h\" (UniqueName: \"kubernetes.io/projected/2cb74db0-df6e-471c-bce4-7de34b2a33e3-kube-api-access-4qq9h\") pod \"e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.291198 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.291048 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:32.348394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.348354 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:32.733152 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.733122 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw"] Oct 30 12:26:32.734797 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:26:32.734770 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb74db0_df6e_471c_bce4_7de34b2a33e3.slice/crio-37c50fd8465ff9bb82c8c6dfbabbaaed6e418e325ec27498c51482a297af4e24 WatchSource:0}: Error finding container 37c50fd8465ff9bb82c8c6dfbabbaaed6e418e325ec27498c51482a297af4e24: Status 404 returned error can't find the container with id 37c50fd8465ff9bb82c8c6dfbabbaaed6e418e325ec27498c51482a297af4e24 Oct 30 12:26:32.738482 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.738437 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:26:32.738573 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.738537 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:26:32.738612 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:32.738580 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:26:33.092158 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:33.092055 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" event={"ID":"2cb74db0-df6e-471c-bce4-7de34b2a33e3","Type":"ContainerStarted","Data":"9653960bc2c2c48634dec6cb66996a6b53430357553c14e919ec81dbf4a4d690"} Oct 30 12:26:33.092158 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:33.092116 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" event={"ID":"2cb74db0-df6e-471c-bce4-7de34b2a33e3","Type":"ContainerStarted","Data":"37c50fd8465ff9bb82c8c6dfbabbaaed6e418e325ec27498c51482a297af4e24"} Oct 30 12:26:33.349428 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:33.349340 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:33.355322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:33.355294 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:33.400834 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:33.400772 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" podStartSLOduration=2.400758033 podStartE2EDuration="2.400758033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:33.149483235 +0000 UTC m=+1946.259326117" watchObservedRunningTime="2026-10-30 12:26:33.400758033 +0000 UTC m=+1946.510600914" Oct 30 12:26:33.592782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:33.592737 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:33.768442 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:33.768409 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:26:33.768596 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:33.768469 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" podUID="5235ef15-0da6-4593-85a6-8d453b249064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:26:34.098792 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.098717 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:34.099715 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.099695 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:26:34.727676 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.727646 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:26:34.779467 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.779429 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-workload-certs\") pod \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " Oct 30 12:26:34.779667 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.779516 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-credential-socket\") pod \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " Oct 30 12:26:34.779667 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.779566 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-token\" (UniqueName: \"kubernetes.io/projected/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-token\") pod \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " Oct 30 12:26:34.779782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.779713 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istiod-ca-cert\") pod \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " Oct 30 12:26:34.779782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.779713 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-workload-certs" pod "67aa5abb-dd1e-4ac9-888e-bb958fddffa9" (UID: "67aa5abb-dd1e-4ac9-888e-bb958fddffa9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:34.779782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.779754 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-data\") pod \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " Oct 30 12:26:34.779984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.779844 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-credential-socket" pod "67aa5abb-dd1e-4ac9-888e-bb958fddffa9" (UID: "67aa5abb-dd1e-4ac9-888e-bb958fddffa9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:34.779984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.779876 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-workload-socket\") pod \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " Oct 30 12:26:34.779984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.779933 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-kube-api-access-kbfdp\" (UniqueName: \"kubernetes.io/projected/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-kube-api-access-kbfdp\") pod \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " Oct 30 12:26:34.780136 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.779983 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-podinfo\") pod \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " Oct 30 12:26:34.780136 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.780021 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-envoy\") pod \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\" (UID: \"67aa5abb-dd1e-4ac9-888e-bb958fddffa9\") " Oct 30 12:26:34.780136 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.780067 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-data" pod "67aa5abb-dd1e-4ac9-888e-bb958fddffa9" (UID: "67aa5abb-dd1e-4ac9-888e-bb958fddffa9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:34.780136 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.780102 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istiod-ca-cert" pod "67aa5abb-dd1e-4ac9-888e-bb958fddffa9" (UID: "67aa5abb-dd1e-4ac9-888e-bb958fddffa9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:26:34.780383 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.780200 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-workload-socket" pod "67aa5abb-dd1e-4ac9-888e-bb958fddffa9" (UID: "67aa5abb-dd1e-4ac9-888e-bb958fddffa9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:34.780856 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.780830 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:34.780985 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.780858 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:34.780985 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.780875 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:34.780985 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.780890 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:34.780985 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.780905 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:34.782902 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.782878 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-envoy" pod "67aa5abb-dd1e-4ac9-888e-bb958fddffa9" (UID: "67aa5abb-dd1e-4ac9-888e-bb958fddffa9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:26:34.783158 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.783131 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-podinfo" pod "67aa5abb-dd1e-4ac9-888e-bb958fddffa9" (UID: "67aa5abb-dd1e-4ac9-888e-bb958fddffa9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:26:34.784137 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.784021 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-kube-api-access-kbfdp" pod "67aa5abb-dd1e-4ac9-888e-bb958fddffa9" (UID: "67aa5abb-dd1e-4ac9-888e-bb958fddffa9"). InnerVolumeSpecName "kube-api-access-kbfdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:26:34.784137 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.784113 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-token" pod "67aa5abb-dd1e-4ac9-888e-bb958fddffa9" (UID: "67aa5abb-dd1e-4ac9-888e-bb958fddffa9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:26:34.882503 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.882463 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbfdp\" (UniqueName: \"kubernetes.io/projected/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-kube-api-access-kbfdp\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:34.882503 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.882497 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:34.882503 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.882507 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:34.882736 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:34.882517 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aa5abb-dd1e-4ac9-888e-bb958fddffa9-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:35.109803 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:35.109764 4413 generic.go:356] "Generic (PLEG): container finished" podID="67aa5abb-dd1e-4ac9-888e-bb958fddffa9" containerID="762ee8d14c218d84d114435177787d40ac93425aecb31ee70027fee69949bf42" exitCode=0 Oct 30 12:26:35.110353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:35.109827 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" Oct 30 12:26:35.110353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:35.109858 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" event={"ID":"67aa5abb-dd1e-4ac9-888e-bb958fddffa9","Type":"ContainerDied","Data":"762ee8d14c218d84d114435177787d40ac93425aecb31ee70027fee69949bf42"} Oct 30 12:26:35.110353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:35.109931 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz" event={"ID":"67aa5abb-dd1e-4ac9-888e-bb958fddffa9","Type":"ContainerDied","Data":"a4e97fddc13ec7a61a943824c8b14ca9978f2f4767b71e388dfc1f4770f9f88b"} Oct 30 12:26:35.110353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:35.109960 4413 scope.go:122] "RemoveContainer" containerID="762ee8d14c218d84d114435177787d40ac93425aecb31ee70027fee69949bf42" Oct 30 12:26:35.122131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:35.122108 4413 scope.go:122] "RemoveContainer" containerID="06dba9e671558e66e3ce60482f497730e5a6067fdf2cf9a225ea42e53aa2f5e4" Oct 30 12:26:35.134945 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:35.134918 4413 scope.go:122] "RemoveContainer" containerID="762ee8d14c218d84d114435177787d40ac93425aecb31ee70027fee69949bf42" Oct 30 12:26:35.135298 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:26:35.135278 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762ee8d14c218d84d114435177787d40ac93425aecb31ee70027fee69949bf42\": container with ID starting with 762ee8d14c218d84d114435177787d40ac93425aecb31ee70027fee69949bf42 not found: ID does not exist" containerID="762ee8d14c218d84d114435177787d40ac93425aecb31ee70027fee69949bf42" Oct 30 12:26:35.135379 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:35.135307 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762ee8d14c218d84d114435177787d40ac93425aecb31ee70027fee69949bf42"} err="failed to get container status \"762ee8d14c218d84d114435177787d40ac93425aecb31ee70027fee69949bf42\": rpc error: code = NotFound desc = could not find container \"762ee8d14c218d84d114435177787d40ac93425aecb31ee70027fee69949bf42\": container with ID starting with 762ee8d14c218d84d114435177787d40ac93425aecb31ee70027fee69949bf42 not found: ID does not exist" Oct 30 12:26:35.135379 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:35.135323 4413 scope.go:122] "RemoveContainer" containerID="06dba9e671558e66e3ce60482f497730e5a6067fdf2cf9a225ea42e53aa2f5e4" Oct 30 12:26:35.135619 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:26:35.135602 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06dba9e671558e66e3ce60482f497730e5a6067fdf2cf9a225ea42e53aa2f5e4\": container with ID starting with 06dba9e671558e66e3ce60482f497730e5a6067fdf2cf9a225ea42e53aa2f5e4 not found: ID does not exist" containerID="06dba9e671558e66e3ce60482f497730e5a6067fdf2cf9a225ea42e53aa2f5e4" Oct 30 12:26:35.135668 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:35.135625 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06dba9e671558e66e3ce60482f497730e5a6067fdf2cf9a225ea42e53aa2f5e4"} err="failed to get container status \"06dba9e671558e66e3ce60482f497730e5a6067fdf2cf9a225ea42e53aa2f5e4\": rpc error: code = NotFound desc = could not find container \"06dba9e671558e66e3ce60482f497730e5a6067fdf2cf9a225ea42e53aa2f5e4\": container with ID starting with 06dba9e671558e66e3ce60482f497730e5a6067fdf2cf9a225ea42e53aa2f5e4 not found: ID does not exist" Oct 30 12:26:35.163515 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:35.163486 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz"] Oct 30 12:26:35.175111 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:35.175086 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-922643-openshift-default-9547f9f9c-2qbwz"] Oct 30 12:26:35.405504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:35.405466 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:35.679018 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:35.678948 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:35.794845 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:35.794817 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:35.816148 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:26:35.816114 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:26:36.120389 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:36.120287 4413 generic.go:356] "Generic (PLEG): container finished" podID="5235ef15-0da6-4593-85a6-8d453b249064" containerID="c4d0454bb4d05d0982eb72d09adb224e1b67f6f1df01d0a62beb452ff10621e3" exitCode=0 Oct 30 12:26:36.120389 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:36.120359 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" event={"ID":"5235ef15-0da6-4593-85a6-8d453b249064","Type":"ContainerDied","Data":"c4d0454bb4d05d0982eb72d09adb224e1b67f6f1df01d0a62beb452ff10621e3"} Oct 30 12:26:36.120998 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:36.120981 4413 scope.go:122] "RemoveContainer" containerID="c4d0454bb4d05d0982eb72d09adb224e1b67f6f1df01d0a62beb452ff10621e3" Oct 30 12:26:36.122726 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:36.122697 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:26:36.122823 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:36.122766 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:26:36.122823 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:36.122800 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:26:37.026760 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.026730 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67aa5abb-dd1e-4ac9-888e-bb958fddffa9" path="/var/lib/kubelet/pods/67aa5abb-dd1e-4ac9-888e-bb958fddffa9/volumes" Oct 30 12:26:37.132097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.132062 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" event={"ID":"5235ef15-0da6-4593-85a6-8d453b249064","Type":"ContainerStarted","Data":"375ce1a6d4ff55680c5015a61d1cf076bd60bf0d7a300deb3ce7a64c61645190"} Oct 30 12:26:37.250716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.250688 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:37.436173 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.436143 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:37.742721 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.742635 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s"] Oct 30 12:26:37.749523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.749496 4413 cpu_manager.go:414] "containerMap: removing container" podUID="67aa5abb-dd1e-4ac9-888e-bb958fddffa9" containerName="istio-proxy" Oct 30 12:26:37.749523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.749523 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67aa5abb-dd1e-4ac9-888e-bb958fddffa9" containerName="istio-proxy" Oct 30 12:26:37.749760 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.749579 4413 cpu_manager.go:414] "containerMap: removing container" podUID="67aa5abb-dd1e-4ac9-888e-bb958fddffa9" containerName="istio-proxy" Oct 30 12:26:37.749760 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.749589 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67aa5abb-dd1e-4ac9-888e-bb958fddffa9" containerName="istio-proxy" Oct 30 12:26:37.750353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.750332 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="67aa5abb-dd1e-4ac9-888e-bb958fddffa9" containerName="istio-proxy" Oct 30 12:26:37.763628 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.763602 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s"] Oct 30 12:26:37.763805 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.763708 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s" Oct 30 12:26:37.779675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.779653 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5c57af\"" Oct 30 12:26:37.851795 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.851770 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5c57af-dockercfg-hg5dn\"" Oct 30 12:26:37.861581 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.861555 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e97203-7545-4a2d-9382-3804e526c483-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s\" (UID: \"f3e97203-7545-4a2d-9382-3804e526c483\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s" Oct 30 12:26:37.861715 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.861672 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtmsh\" (UniqueName: \"kubernetes.io/projected/f3e97203-7545-4a2d-9382-3804e526c483-kube-api-access-vtmsh\") pod \"payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s\" (UID: \"f3e97203-7545-4a2d-9382-3804e526c483\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s" Oct 30 12:26:37.865913 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.865894 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:26:37.866113 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.866093 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 12:26:37.866153 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.866139 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" podUID="5235ef15-0da6-4593-85a6-8d453b249064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 12:26:37.963002 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.962971 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e97203-7545-4a2d-9382-3804e526c483-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s\" (UID: \"f3e97203-7545-4a2d-9382-3804e526c483\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s" Oct 30 12:26:37.963156 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.963059 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtmsh\" (UniqueName: \"kubernetes.io/projected/f3e97203-7545-4a2d-9382-3804e526c483-kube-api-access-vtmsh\") pod \"payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s\" (UID: \"f3e97203-7545-4a2d-9382-3804e526c483\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s" Oct 30 12:26:37.963717 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:37.963695 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e97203-7545-4a2d-9382-3804e526c483-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s\" (UID: \"f3e97203-7545-4a2d-9382-3804e526c483\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s" Oct 30 12:26:38.034551 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:38.034483 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtmsh\" (UniqueName: \"kubernetes.io/projected/f3e97203-7545-4a2d-9382-3804e526c483-kube-api-access-vtmsh\") pod \"payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s\" (UID: \"f3e97203-7545-4a2d-9382-3804e526c483\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s" Oct 30 12:26:38.078597 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:38.078569 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s" Oct 30 12:26:38.419364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:38.419334 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s"] Oct 30 12:26:38.420603 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:26:38.420564 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e97203_7545_4a2d_9382_3804e526c483.slice/crio-938c919cd232581fd049634d0402878f33245e8e6e864a79d7e4bf8e21917e2f WatchSource:0}: Error finding container 938c919cd232581fd049634d0402878f33245e8e6e864a79d7e4bf8e21917e2f: Status 404 returned error can't find the container with id 938c919cd232581fd049634d0402878f33245e8e6e864a79d7e4bf8e21917e2f Oct 30 12:26:38.866107 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:38.866026 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 12:26:38.866107 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:38.866082 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" podUID="5235ef15-0da6-4593-85a6-8d453b249064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 12:26:39.041573 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:39.041535 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:39.153866 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:39.153834 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s" event={"ID":"f3e97203-7545-4a2d-9382-3804e526c483","Type":"ContainerStarted","Data":"595c9162c031a33d9e0b3875237f5c7a41b70e82add03aefa767825f830f03b4"} Oct 30 12:26:39.154087 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:39.153887 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s" Oct 30 12:26:39.154087 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:39.153909 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s" event={"ID":"f3e97203-7545-4a2d-9382-3804e526c483","Type":"ContainerStarted","Data":"938c919cd232581fd049634d0402878f33245e8e6e864a79d7e4bf8e21917e2f"} Oct 30 12:26:39.214272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:39.213982 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s" podStartSLOduration=2.213962626 podStartE2EDuration="2.213962626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:39.212041158 +0000 UTC m=+1952.321884066" watchObservedRunningTime="2026-10-30 12:26:39.213962626 +0000 UTC m=+1952.323805507" Oct 30 12:26:39.866075 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:39.866043 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 12:26:39.866488 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:39.866093 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" podUID="5235ef15-0da6-4593-85a6-8d453b249064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 12:26:40.865788 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:40.865750 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 12:26:40.865990 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:40.865802 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" podUID="5235ef15-0da6-4593-85a6-8d453b249064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 12:26:40.911273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:40.900737 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:41.276360 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:41.276326 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt"] Oct 30 12:26:41.385415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:41.385374 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:41.529775 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:41.529685 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:41.866104 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:41.866022 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 12:26:41.866104 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:41.866085 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" podUID="5235ef15-0da6-4593-85a6-8d453b249064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 12:26:42.869591 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:42.869553 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:26:42.870050 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:42.869853 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:26:42.870606 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:42.870591 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:26:43.356070 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:43.356033 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:44.425804 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:44.425773 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24"] Oct 30 12:26:44.461830 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:44.461802 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv"] Oct 30 12:26:44.567996 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:44.567945 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:44.695924 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:44.695820 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:44.835118 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:44.835085 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:45.332680 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:45.332651 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:46.276891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:46.276854 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt" podUID="a488bb31-f27f-4d15-85a1-f11468ea3fec" containerName="payload-pre-processing" containerID="cri-o://6f81a6b45cfac308addb37fd9dcd173aa54888f2a34ca63c7a4b313056a22cbd" gracePeriod=25 Oct 30 12:26:46.549474 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:46.549451 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt" Oct 30 12:26:46.627148 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:46.627117 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a488bb31-f27f-4d15-85a1-f11468ea3fec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a488bb31-f27f-4d15-85a1-f11468ea3fec-plugins-config-volume\") pod \"a488bb31-f27f-4d15-85a1-f11468ea3fec\" (UID: \"a488bb31-f27f-4d15-85a1-f11468ea3fec\") " Oct 30 12:26:46.627416 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:46.627397 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a488bb31-f27f-4d15-85a1-f11468ea3fec-kube-api-access-wbq5l\" (UniqueName: \"kubernetes.io/projected/a488bb31-f27f-4d15-85a1-f11468ea3fec-kube-api-access-wbq5l\") pod \"a488bb31-f27f-4d15-85a1-f11468ea3fec\" (UID: \"a488bb31-f27f-4d15-85a1-f11468ea3fec\") " Oct 30 12:26:46.627619 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:46.627585 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a488bb31-f27f-4d15-85a1-f11468ea3fec-plugins-config-volume" pod "a488bb31-f27f-4d15-85a1-f11468ea3fec" (UID: "a488bb31-f27f-4d15-85a1-f11468ea3fec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:26:46.628006 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:46.627980 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a488bb31-f27f-4d15-85a1-f11468ea3fec-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:46.629742 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:46.629712 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a488bb31-f27f-4d15-85a1-f11468ea3fec-kube-api-access-wbq5l" pod "a488bb31-f27f-4d15-85a1-f11468ea3fec" (UID: "a488bb31-f27f-4d15-85a1-f11468ea3fec"). InnerVolumeSpecName "kube-api-access-wbq5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:26:46.729636 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:46.729600 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbq5l\" (UniqueName: \"kubernetes.io/projected/a488bb31-f27f-4d15-85a1-f11468ea3fec-kube-api-access-wbq5l\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:47.219648 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:47.219617 4413 generic.go:356] "Generic (PLEG): container finished" podID="a488bb31-f27f-4d15-85a1-f11468ea3fec" containerID="6f81a6b45cfac308addb37fd9dcd173aa54888f2a34ca63c7a4b313056a22cbd" exitCode=0 Oct 30 12:26:47.219807 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:47.219667 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt" Oct 30 12:26:47.219807 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:47.219700 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt" event={"ID":"a488bb31-f27f-4d15-85a1-f11468ea3fec","Type":"ContainerDied","Data":"6f81a6b45cfac308addb37fd9dcd173aa54888f2a34ca63c7a4b313056a22cbd"} Oct 30 12:26:47.219807 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:47.219762 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt" event={"ID":"a488bb31-f27f-4d15-85a1-f11468ea3fec","Type":"ContainerDied","Data":"fe13d1d3025cffd2c6c5f0e50e03e4db93336826fd50d9db74014a218c079311"} Oct 30 12:26:47.219807 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:47.219794 4413 scope.go:122] "RemoveContainer" containerID="6f81a6b45cfac308addb37fd9dcd173aa54888f2a34ca63c7a4b313056a22cbd" Oct 30 12:26:47.229703 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:47.229676 4413 scope.go:122] "RemoveContainer" containerID="6f81a6b45cfac308addb37fd9dcd173aa54888f2a34ca63c7a4b313056a22cbd" Oct 30 12:26:47.230002 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:26:47.229984 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f81a6b45cfac308addb37fd9dcd173aa54888f2a34ca63c7a4b313056a22cbd\": container with ID starting with 6f81a6b45cfac308addb37fd9dcd173aa54888f2a34ca63c7a4b313056a22cbd not found: ID does not exist" containerID="6f81a6b45cfac308addb37fd9dcd173aa54888f2a34ca63c7a4b313056a22cbd" Oct 30 12:26:47.230061 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:47.230013 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f81a6b45cfac308addb37fd9dcd173aa54888f2a34ca63c7a4b313056a22cbd"} err="failed to get container status \"6f81a6b45cfac308addb37fd9dcd173aa54888f2a34ca63c7a4b313056a22cbd\": rpc error: code = NotFound desc = could not find container \"6f81a6b45cfac308addb37fd9dcd173aa54888f2a34ca63c7a4b313056a22cbd\": container with ID starting with 6f81a6b45cfac308addb37fd9dcd173aa54888f2a34ca63c7a4b313056a22cbd not found: ID does not exist" Oct 30 12:26:47.334389 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:47.334301 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt"] Oct 30 12:26:47.349784 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:47.349753 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aef61e-74d8ddf95-lgjnt"] Oct 30 12:26:47.545529 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:47.545494 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:49.020491 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:49.020453 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a488bb31-f27f-4d15-85a1-f11468ea3fec" path="/var/lib/kubelet/pods/a488bb31-f27f-4d15-85a1-f11468ea3fec/volumes" Oct 30 12:26:49.426405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:49.426330 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24" podUID="d0994ac6-f047-4b32-ad6b-acad3336a059" containerName="payload-processing" containerID="cri-o://0aab0c35802d9bac23a1a3630a1528713249714f82e8ecf05d0f7930fd2cc79d" gracePeriod=25 Oct 30 12:26:49.462243 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:49.462203 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv" podUID="edf395c4-4970-4487-9952-fab1b6c07f6a" containerName="payload-pre-processing" containerID="cri-o://16a56cf34935c864857d9dd0e0ea03c695a9b87c7c87e03c9d45f6d2292e943b" gracePeriod=25 Oct 30 12:26:49.745183 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:49.745155 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24" Oct 30 12:26:49.797219 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:49.797192 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0994ac6-f047-4b32-ad6b-acad3336a059-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0994ac6-f047-4b32-ad6b-acad3336a059-plugins-config-volume\") pod \"d0994ac6-f047-4b32-ad6b-acad3336a059\" (UID: \"d0994ac6-f047-4b32-ad6b-acad3336a059\") " Oct 30 12:26:49.797378 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:49.797325 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0994ac6-f047-4b32-ad6b-acad3336a059-kube-api-access-dqc7h\" (UniqueName: \"kubernetes.io/projected/d0994ac6-f047-4b32-ad6b-acad3336a059-kube-api-access-dqc7h\") pod \"d0994ac6-f047-4b32-ad6b-acad3336a059\" (UID: \"d0994ac6-f047-4b32-ad6b-acad3336a059\") " Oct 30 12:26:49.797603 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:49.797577 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0994ac6-f047-4b32-ad6b-acad3336a059-plugins-config-volume" pod "d0994ac6-f047-4b32-ad6b-acad3336a059" (UID: "d0994ac6-f047-4b32-ad6b-acad3336a059"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:26:49.798015 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:49.797994 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv" Oct 30 12:26:49.798137 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:49.798046 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0994ac6-f047-4b32-ad6b-acad3336a059-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:49.799613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:49.799588 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0994ac6-f047-4b32-ad6b-acad3336a059-kube-api-access-dqc7h" pod "d0994ac6-f047-4b32-ad6b-acad3336a059" (UID: "d0994ac6-f047-4b32-ad6b-acad3336a059"). InnerVolumeSpecName "kube-api-access-dqc7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:26:49.899051 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:49.899016 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edf395c4-4970-4487-9952-fab1b6c07f6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edf395c4-4970-4487-9952-fab1b6c07f6a-plugins-config-volume\") pod \"edf395c4-4970-4487-9952-fab1b6c07f6a\" (UID: \"edf395c4-4970-4487-9952-fab1b6c07f6a\") " Oct 30 12:26:49.899238 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:49.899104 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf395c4-4970-4487-9952-fab1b6c07f6a-kube-api-access-pwhs5\" (UniqueName: \"kubernetes.io/projected/edf395c4-4970-4487-9952-fab1b6c07f6a-kube-api-access-pwhs5\") pod \"edf395c4-4970-4487-9952-fab1b6c07f6a\" (UID: \"edf395c4-4970-4487-9952-fab1b6c07f6a\") " Oct 30 12:26:49.899427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:49.899403 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf395c4-4970-4487-9952-fab1b6c07f6a-plugins-config-volume" pod "edf395c4-4970-4487-9952-fab1b6c07f6a" (UID: "edf395c4-4970-4487-9952-fab1b6c07f6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:26:49.899691 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:49.899673 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edf395c4-4970-4487-9952-fab1b6c07f6a-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:49.899755 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:49.899697 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqc7h\" (UniqueName: \"kubernetes.io/projected/d0994ac6-f047-4b32-ad6b-acad3336a059-kube-api-access-dqc7h\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:49.901296 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:49.901275 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf395c4-4970-4487-9952-fab1b6c07f6a-kube-api-access-pwhs5" pod "edf395c4-4970-4487-9952-fab1b6c07f6a" (UID: "edf395c4-4970-4487-9952-fab1b6c07f6a"). InnerVolumeSpecName "kube-api-access-pwhs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:26:50.000797 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.000724 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwhs5\" (UniqueName: \"kubernetes.io/projected/edf395c4-4970-4487-9952-fab1b6c07f6a-kube-api-access-pwhs5\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:50.164691 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.164663 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s" Oct 30 12:26:50.242298 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.242235 4413 generic.go:356] "Generic (PLEG): container finished" podID="edf395c4-4970-4487-9952-fab1b6c07f6a" containerID="16a56cf34935c864857d9dd0e0ea03c695a9b87c7c87e03c9d45f6d2292e943b" exitCode=0 Oct 30 12:26:50.242298 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.242288 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv" event={"ID":"edf395c4-4970-4487-9952-fab1b6c07f6a","Type":"ContainerDied","Data":"16a56cf34935c864857d9dd0e0ea03c695a9b87c7c87e03c9d45f6d2292e943b"} Oct 30 12:26:50.242298 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.242301 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv" Oct 30 12:26:50.242603 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.242343 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv" event={"ID":"edf395c4-4970-4487-9952-fab1b6c07f6a","Type":"ContainerDied","Data":"76844d587545dd678ac5cff2f4c99a241f03c7699a4dad2c3f1cc8ba2aaf03fd"} Oct 30 12:26:50.242603 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.242371 4413 scope.go:122] "RemoveContainer" containerID="16a56cf34935c864857d9dd0e0ea03c695a9b87c7c87e03c9d45f6d2292e943b" Oct 30 12:26:50.243892 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.243864 4413 generic.go:356] "Generic (PLEG): container finished" podID="d0994ac6-f047-4b32-ad6b-acad3336a059" containerID="0aab0c35802d9bac23a1a3630a1528713249714f82e8ecf05d0f7930fd2cc79d" exitCode=0 Oct 30 12:26:50.244033 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.243917 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24" Oct 30 12:26:50.244033 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.243939 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24" event={"ID":"d0994ac6-f047-4b32-ad6b-acad3336a059","Type":"ContainerDied","Data":"0aab0c35802d9bac23a1a3630a1528713249714f82e8ecf05d0f7930fd2cc79d"} Oct 30 12:26:50.244033 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.243994 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24" event={"ID":"d0994ac6-f047-4b32-ad6b-acad3336a059","Type":"ContainerDied","Data":"b101572ec1b876562d737e1391bb4194ceb78aaddaa2936389bb17a377e9d4dd"} Oct 30 12:26:50.253657 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.253633 4413 scope.go:122] "RemoveContainer" containerID="16a56cf34935c864857d9dd0e0ea03c695a9b87c7c87e03c9d45f6d2292e943b" Oct 30 12:26:50.253986 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:26:50.253961 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a56cf34935c864857d9dd0e0ea03c695a9b87c7c87e03c9d45f6d2292e943b\": container with ID starting with 16a56cf34935c864857d9dd0e0ea03c695a9b87c7c87e03c9d45f6d2292e943b not found: ID does not exist" containerID="16a56cf34935c864857d9dd0e0ea03c695a9b87c7c87e03c9d45f6d2292e943b" Oct 30 12:26:50.254063 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.253997 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a56cf34935c864857d9dd0e0ea03c695a9b87c7c87e03c9d45f6d2292e943b"} err="failed to get container status \"16a56cf34935c864857d9dd0e0ea03c695a9b87c7c87e03c9d45f6d2292e943b\": rpc error: code = NotFound desc = could not find container \"16a56cf34935c864857d9dd0e0ea03c695a9b87c7c87e03c9d45f6d2292e943b\": container with ID starting with 16a56cf34935c864857d9dd0e0ea03c695a9b87c7c87e03c9d45f6d2292e943b not found: ID does not exist" Oct 30 12:26:50.254063 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.254015 4413 scope.go:122] "RemoveContainer" containerID="0aab0c35802d9bac23a1a3630a1528713249714f82e8ecf05d0f7930fd2cc79d" Oct 30 12:26:50.264937 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.264915 4413 scope.go:122] "RemoveContainer" containerID="0aab0c35802d9bac23a1a3630a1528713249714f82e8ecf05d0f7930fd2cc79d" Oct 30 12:26:50.265217 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:26:50.265197 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aab0c35802d9bac23a1a3630a1528713249714f82e8ecf05d0f7930fd2cc79d\": container with ID starting with 0aab0c35802d9bac23a1a3630a1528713249714f82e8ecf05d0f7930fd2cc79d not found: ID does not exist" containerID="0aab0c35802d9bac23a1a3630a1528713249714f82e8ecf05d0f7930fd2cc79d" Oct 30 12:26:50.265320 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.265221 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aab0c35802d9bac23a1a3630a1528713249714f82e8ecf05d0f7930fd2cc79d"} err="failed to get container status \"0aab0c35802d9bac23a1a3630a1528713249714f82e8ecf05d0f7930fd2cc79d\": rpc error: code = NotFound desc = could not find container \"0aab0c35802d9bac23a1a3630a1528713249714f82e8ecf05d0f7930fd2cc79d\": container with ID starting with 0aab0c35802d9bac23a1a3630a1528713249714f82e8ecf05d0f7930fd2cc79d not found: ID does not exist" Oct 30 12:26:50.298475 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.298410 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:50.317410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.317381 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv"] Oct 30 12:26:50.321533 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.321506 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-431bf6-555466f5f5-4dbmv"] Oct 30 12:26:50.360285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.360237 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24"] Oct 30 12:26:50.368728 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:50.368700 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-431bf6-698c4475f4-w2q24"] Oct 30 12:26:51.020546 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:51.020514 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0994ac6-f047-4b32-ad6b-acad3336a059" path="/var/lib/kubelet/pods/d0994ac6-f047-4b32-ad6b-acad3336a059/volumes" Oct 30 12:26:51.020913 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:51.020900 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edf395c4-4970-4487-9952-fab1b6c07f6a" path="/var/lib/kubelet/pods/edf395c4-4970-4487-9952-fab1b6c07f6a/volumes" Oct 30 12:26:51.517595 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:51.517567 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z"] Oct 30 12:26:51.767488 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:51.767453 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:51.892982 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:51.892930 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:52.071842 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:52.071789 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:53.731670 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.731631 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv"] Oct 30 12:26:53.732500 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.732479 4413 cpu_manager.go:414] "containerMap: removing container" podUID="a488bb31-f27f-4d15-85a1-f11468ea3fec" containerName="payload-pre-processing" Oct 30 12:26:53.732617 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.732501 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a488bb31-f27f-4d15-85a1-f11468ea3fec" containerName="payload-pre-processing" Oct 30 12:26:53.732617 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.732549 4413 cpu_manager.go:414] "containerMap: removing container" podUID="d0994ac6-f047-4b32-ad6b-acad3336a059" containerName="payload-processing" Oct 30 12:26:53.732617 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.732557 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0994ac6-f047-4b32-ad6b-acad3336a059" containerName="payload-processing" Oct 30 12:26:53.732617 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.732583 4413 cpu_manager.go:414] "containerMap: removing container" podUID="edf395c4-4970-4487-9952-fab1b6c07f6a" containerName="payload-pre-processing" Oct 30 12:26:53.732617 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.732592 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf395c4-4970-4487-9952-fab1b6c07f6a" containerName="payload-pre-processing" Oct 30 12:26:53.732873 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.732777 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="a488bb31-f27f-4d15-85a1-f11468ea3fec" containerName="payload-pre-processing" Oct 30 12:26:53.732873 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.732793 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="67aa5abb-dd1e-4ac9-888e-bb958fddffa9" containerName="istio-proxy" Oct 30 12:26:53.732873 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.732811 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0994ac6-f047-4b32-ad6b-acad3336a059" containerName="payload-processing" Oct 30 12:26:53.732873 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.732823 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="edf395c4-4970-4487-9952-fab1b6c07f6a" containerName="payload-pre-processing" Oct 30 12:26:53.738205 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.738180 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" Oct 30 12:26:53.757317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.757279 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-096834-serving-cert\"" Oct 30 12:26:53.774366 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.774319 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv"] Oct 30 12:26:53.860173 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.860139 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cca05f66-33b8-4d90-89d1-b9b7559fcc32-metrics-certs\") pod \"maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv\" (UID: \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" Oct 30 12:26:53.860387 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.860179 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b927\" (UniqueName: \"kubernetes.io/projected/cca05f66-33b8-4d90-89d1-b9b7559fcc32-kube-api-access-6b927\") pod \"maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv\" (UID: \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" Oct 30 12:26:53.860462 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.860442 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cca05f66-33b8-4d90-89d1-b9b7559fcc32-maas-api-tls\") pod \"maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv\" (UID: \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" Oct 30 12:26:53.860505 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.860475 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca05f66-33b8-4d90-89d1-b9b7559fcc32-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv\" (UID: \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" Oct 30 12:26:53.874113 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.874063 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:53.967896 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.967864 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cca05f66-33b8-4d90-89d1-b9b7559fcc32-maas-api-tls\") pod \"maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv\" (UID: \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" Oct 30 12:26:53.968380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.968355 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca05f66-33b8-4d90-89d1-b9b7559fcc32-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv\" (UID: \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" Oct 30 12:26:53.970551 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.969281 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca05f66-33b8-4d90-89d1-b9b7559fcc32-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv\" (UID: \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" Oct 30 12:26:53.970853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.970831 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cca05f66-33b8-4d90-89d1-b9b7559fcc32-metrics-certs\") pod \"maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv\" (UID: \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" Oct 30 12:26:53.971571 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.971546 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b927\" (UniqueName: \"kubernetes.io/projected/cca05f66-33b8-4d90-89d1-b9b7559fcc32-kube-api-access-6b927\") pod \"maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv\" (UID: \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" Oct 30 12:26:53.983941 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.983360 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cca05f66-33b8-4d90-89d1-b9b7559fcc32-metrics-certs\") pod \"maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv\" (UID: \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" Oct 30 12:26:53.983941 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:53.983822 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cca05f66-33b8-4d90-89d1-b9b7559fcc32-maas-api-tls\") pod \"maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv\" (UID: \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" Oct 30 12:26:54.013273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:54.011337 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b927\" (UniqueName: \"kubernetes.io/projected/cca05f66-33b8-4d90-89d1-b9b7559fcc32-kube-api-access-6b927\") pod \"maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv\" (UID: \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" Oct 30 12:26:54.018052 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:54.018021 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:54.078527 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:54.078504 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" Oct 30 12:26:54.417911 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:54.417885 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv"] Oct 30 12:26:54.419972 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:26:54.419944 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca05f66_33b8_4d90_89d1_b9b7559fcc32.slice/crio-20a82417903e87968274f66bfb63f5ee32863bd5a4648757bd13d4b07b4721de WatchSource:0}: Error finding container 20a82417903e87968274f66bfb63f5ee32863bd5a4648757bd13d4b07b4721de: Status 404 returned error can't find the container with id 20a82417903e87968274f66bfb63f5ee32863bd5a4648757bd13d4b07b4721de Oct 30 12:26:55.342333 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:55.342213 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" event={"ID":"cca05f66-33b8-4d90-89d1-b9b7559fcc32","Type":"ContainerStarted","Data":"231d5019d84208c748efb31329ce05b61d6a2a760d8bc5488c3c7614fb8a5916"} Oct 30 12:26:55.342892 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:55.342860 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" event={"ID":"cca05f66-33b8-4d90-89d1-b9b7559fcc32","Type":"ContainerStarted","Data":"20a82417903e87968274f66bfb63f5ee32863bd5a4648757bd13d4b07b4721de"} Oct 30 12:26:55.388593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:55.388543 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" podStartSLOduration=1.839465117 podStartE2EDuration="2.388529374s" totalImagesPullingTime="549.064257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:55.385566014 +0000 UTC m=+1968.495408967" watchObservedRunningTime="2026-10-30 12:26:55.388529374 +0000 UTC m=+1968.498372258" Oct 30 12:26:55.795817 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:55.795777 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:56.348241 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:56.348208 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" Oct 30 12:26:56.518427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:56.518380 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z" podUID="e4d03a58-07e0-429f-bb8b-b3a9a0f09164" containerName="payload-processing" containerID="cri-o://8b06d1233d4983874efc63e50f445758cb616634a2564727558f7bd1b832bc8f" gracePeriod=25 Oct 30 12:26:56.826427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:56.826396 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z" Oct 30 12:26:56.947807 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:56.947720 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d03a58-07e0-429f-bb8b-b3a9a0f09164-kube-api-access-khg29\" (UniqueName: \"kubernetes.io/projected/e4d03a58-07e0-429f-bb8b-b3a9a0f09164-kube-api-access-khg29\") pod \"e4d03a58-07e0-429f-bb8b-b3a9a0f09164\" (UID: \"e4d03a58-07e0-429f-bb8b-b3a9a0f09164\") " Oct 30 12:26:56.947976 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:56.947842 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4d03a58-07e0-429f-bb8b-b3a9a0f09164-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d03a58-07e0-429f-bb8b-b3a9a0f09164-plugins-config-volume\") pod \"e4d03a58-07e0-429f-bb8b-b3a9a0f09164\" (UID: \"e4d03a58-07e0-429f-bb8b-b3a9a0f09164\") " Oct 30 12:26:56.948404 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:56.948374 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d03a58-07e0-429f-bb8b-b3a9a0f09164-plugins-config-volume" pod "e4d03a58-07e0-429f-bb8b-b3a9a0f09164" (UID: "e4d03a58-07e0-429f-bb8b-b3a9a0f09164"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:26:56.948847 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:56.948825 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d03a58-07e0-429f-bb8b-b3a9a0f09164-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:56.950239 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:56.950207 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d03a58-07e0-429f-bb8b-b3a9a0f09164-kube-api-access-khg29" pod "e4d03a58-07e0-429f-bb8b-b3a9a0f09164" (UID: "e4d03a58-07e0-429f-bb8b-b3a9a0f09164"). InnerVolumeSpecName "kube-api-access-khg29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:26:57.050621 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:57.050597 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khg29\" (UniqueName: \"kubernetes.io/projected/e4d03a58-07e0-429f-bb8b-b3a9a0f09164-kube-api-access-khg29\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:26:57.356011 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:57.355928 4413 generic.go:356] "Generic (PLEG): container finished" podID="e4d03a58-07e0-429f-bb8b-b3a9a0f09164" containerID="8b06d1233d4983874efc63e50f445758cb616634a2564727558f7bd1b832bc8f" exitCode=0 Oct 30 12:26:57.356011 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:57.355974 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z" Oct 30 12:26:57.356577 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:57.356009 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z" event={"ID":"e4d03a58-07e0-429f-bb8b-b3a9a0f09164","Type":"ContainerDied","Data":"8b06d1233d4983874efc63e50f445758cb616634a2564727558f7bd1b832bc8f"} Oct 30 12:26:57.356577 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:57.356061 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z" event={"ID":"e4d03a58-07e0-429f-bb8b-b3a9a0f09164","Type":"ContainerDied","Data":"02110652239a3452ba9a0f264b85761f82897efa247406f5f33272e5fca6fd61"} Oct 30 12:26:57.356577 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:57.356090 4413 scope.go:122] "RemoveContainer" containerID="8b06d1233d4983874efc63e50f445758cb616634a2564727558f7bd1b832bc8f" Oct 30 12:26:57.365941 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:57.365926 4413 scope.go:122] "RemoveContainer" containerID="8b06d1233d4983874efc63e50f445758cb616634a2564727558f7bd1b832bc8f" Oct 30 12:26:57.366225 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:26:57.366205 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b06d1233d4983874efc63e50f445758cb616634a2564727558f7bd1b832bc8f\": container with ID starting with 8b06d1233d4983874efc63e50f445758cb616634a2564727558f7bd1b832bc8f not found: ID does not exist" containerID="8b06d1233d4983874efc63e50f445758cb616634a2564727558f7bd1b832bc8f" Oct 30 12:26:57.366322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:57.366261 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b06d1233d4983874efc63e50f445758cb616634a2564727558f7bd1b832bc8f"} err="failed to get container status \"8b06d1233d4983874efc63e50f445758cb616634a2564727558f7bd1b832bc8f\": rpc error: code = NotFound desc = could not find container \"8b06d1233d4983874efc63e50f445758cb616634a2564727558f7bd1b832bc8f\": container with ID starting with 8b06d1233d4983874efc63e50f445758cb616634a2564727558f7bd1b832bc8f not found: ID does not exist" Oct 30 12:26:57.418783 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:57.418753 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z"] Oct 30 12:26:57.428566 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:57.428539 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-266486-796679cbbb-smh2z"] Oct 30 12:26:57.570424 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:57.570396 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:58.101780 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:58.101751 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:26:59.024663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:26:59.024630 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4d03a58-07e0-429f-bb8b-b3a9a0f09164" path="/var/lib/kubelet/pods/e4d03a58-07e0-429f-bb8b-b3a9a0f09164/volumes" Oct 30 12:27:00.378120 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:00.378088 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:02.029782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.029743 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn"] Oct 30 12:27:02.030377 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.030364 4413 cpu_manager.go:414] "containerMap: removing container" podUID="e4d03a58-07e0-429f-bb8b-b3a9a0f09164" containerName="payload-processing" Oct 30 12:27:02.030434 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.030378 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d03a58-07e0-429f-bb8b-b3a9a0f09164" containerName="payload-processing" Oct 30 12:27:02.030504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.030495 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d03a58-07e0-429f-bb8b-b3a9a0f09164" containerName="payload-processing" Oct 30 12:27:02.032906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.032887 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" Oct 30 12:27:02.067344 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.067314 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5c57af-serving-cert\"" Oct 30 12:27:02.072064 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.072032 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn"] Oct 30 12:27:02.126293 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.124378 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn\" (UID: \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" Oct 30 12:27:02.126293 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.124435 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn\" (UID: \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" Oct 30 12:27:02.126293 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.124542 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-metrics-certs\") pod \"maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn\" (UID: \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" Oct 30 12:27:02.126293 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.125234 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng6cf\" (UniqueName: \"kubernetes.io/projected/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-kube-api-access-ng6cf\") pod \"maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn\" (UID: \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" Oct 30 12:27:02.156221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.156181 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:02.227025 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.226990 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng6cf\" (UniqueName: \"kubernetes.io/projected/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-kube-api-access-ng6cf\") pod \"maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn\" (UID: \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" Oct 30 12:27:02.227233 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.227093 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn\" (UID: \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" Oct 30 12:27:02.227233 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.227123 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn\" (UID: \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" Oct 30 12:27:02.227405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.227338 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-metrics-certs\") pod \"maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn\" (UID: \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" Oct 30 12:27:02.227767 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.227748 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn\" (UID: \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" Oct 30 12:27:02.229864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.229843 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-metrics-certs\") pod \"maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn\" (UID: \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" Oct 30 12:27:02.229971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.229933 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn\" (UID: \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" Oct 30 12:27:02.257427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.256957 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng6cf\" (UniqueName: \"kubernetes.io/projected/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-kube-api-access-ng6cf\") pod \"maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn\" (UID: \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" Oct 30 12:27:02.286205 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.286099 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:02.346504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.346473 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" Oct 30 12:27:02.362372 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.362348 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" Oct 30 12:27:02.391767 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.391734 4413 generic.go:356] "Generic (PLEG): container finished" podID="f1e658f2-c95e-40fc-9602-ac8271892d05" containerID="a6b8bd67a912193cde8efa8f34e6233230f257fdc1b0c337753c74af4bdb575c" exitCode=0 Oct 30 12:27:02.391936 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.391802 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" event={"ID":"f1e658f2-c95e-40fc-9602-ac8271892d05","Type":"ContainerDied","Data":"a6b8bd67a912193cde8efa8f34e6233230f257fdc1b0c337753c74af4bdb575c"} Oct 30 12:27:02.392243 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.392229 4413 scope.go:122] "RemoveContainer" containerID="a6b8bd67a912193cde8efa8f34e6233230f257fdc1b0c337753c74af4bdb575c" Oct 30 12:27:02.394689 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.394658 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:27:02.394809 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.394733 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:27:02.394809 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.394773 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:27:02.660432 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:27:02.660383 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bdc2d24_ac84_4bc0_9689_6fa705ccbb2c.slice/crio-7ad75107079cc4d607e8c5c1b122e0ad701b1faf46076b38ec0491b39fa465e8 WatchSource:0}: Error finding container 7ad75107079cc4d607e8c5c1b122e0ad701b1faf46076b38ec0491b39fa465e8: Status 404 returned error can't find the container with id 7ad75107079cc4d607e8c5c1b122e0ad701b1faf46076b38ec0491b39fa465e8 Oct 30 12:27:02.660965 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:02.660930 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn"] Oct 30 12:27:03.431419 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:03.431378 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" event={"ID":"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c","Type":"ContainerStarted","Data":"9609a63cded91a8ce2664f97002640a1b588183459a54a2bbd6f1001029b93be"} Oct 30 12:27:03.444416 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:03.444208 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" event={"ID":"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c","Type":"ContainerStarted","Data":"7ad75107079cc4d607e8c5c1b122e0ad701b1faf46076b38ec0491b39fa465e8"} Oct 30 12:27:03.444416 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:03.444363 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" Oct 30 12:27:03.460316 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:03.454216 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" event={"ID":"f1e658f2-c95e-40fc-9602-ac8271892d05","Type":"ContainerStarted","Data":"6e39776182998c7d2b973c1f0527a13a5d1676cb152a449536abff36457955b5"} Oct 30 12:27:03.516304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:03.515653 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" podStartSLOduration=1.183355631 podStartE2EDuration="1.515633333s" totalImagesPullingTime="332.277702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:27:03.514966633 +0000 UTC m=+1976.624809517" watchObservedRunningTime="2026-10-30 12:27:03.515633333 +0000 UTC m=+1976.625476215" Oct 30 12:27:03.975467 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:03.975428 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:27:03.975673 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:03.975586 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 12:27:03.975673 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:03.975635 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" podUID="f1e658f2-c95e-40fc-9602-ac8271892d05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 12:27:04.321559 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:04.321481 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:04.975052 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:04.975021 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 12:27:04.975505 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:04.975081 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" podUID="f1e658f2-c95e-40fc-9602-ac8271892d05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 12:27:05.837107 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:27:05.837067 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:27:05.975219 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:05.975183 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 12:27:05.975627 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:05.975245 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" podUID="f1e658f2-c95e-40fc-9602-ac8271892d05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 12:27:06.066417 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:06.066371 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:06.974888 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:06.974854 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 12:27:06.975070 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:06.974899 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" podUID="f1e658f2-c95e-40fc-9602-ac8271892d05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 12:27:07.816835 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:07.816740 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:07.974909 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:07.974868 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 12:27:07.975087 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:07.974927 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" podUID="f1e658f2-c95e-40fc-9602-ac8271892d05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 12:27:08.979219 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:08.979189 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:27:08.979715 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:08.979589 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:27:08.980359 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:08.980341 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:27:09.228707 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:09.228666 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:09.386743 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:09.386708 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:09.467804 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:09.467779 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" Oct 30 12:27:10.520440 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:10.520410 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:12.571097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:12.571063 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:13.205506 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:13.205442 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:13.481613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:13.481528 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:13.603861 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:13.603828 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:13.915644 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:13.915610 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:15.277909 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:15.277873 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:16.701470 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:16.701432 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:16.869635 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:16.869602 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:17.701224 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:17.701193 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:17.934661 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:17.934631 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:20.181662 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:20.181629 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:20.309565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:20.309534 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:20.549774 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:20.549674 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:21.743612 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:21.743582 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:22.013576 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:22.013494 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:22.317745 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:22.317661 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:22.497291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:22.493820 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:23.894086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:23.894022 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:25.861933 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:25.861883 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:26.009814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:26.009752 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:26.339917 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:26.339881 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:28.249324 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:28.249197 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:28.380601 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:28.380566 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts"] Oct 30 12:27:28.386928 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:28.386899 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts" Oct 30 12:27:28.421770 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:28.421335 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pshj\" (UniqueName: \"kubernetes.io/projected/b12e3738-51ee-4043-8f3b-f084779b3b9b-kube-api-access-7pshj\") pod \"payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts\" (UID: \"b12e3738-51ee-4043-8f3b-f084779b3b9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts" Oct 30 12:27:28.421770 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:28.421431 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12e3738-51ee-4043-8f3b-f084779b3b9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts\" (UID: \"b12e3738-51ee-4043-8f3b-f084779b3b9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts" Oct 30 12:27:28.429292 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:28.426878 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts"] Oct 30 12:27:28.453056 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:28.452337 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:28.460861 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:28.460830 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7eee3b\"" Oct 30 12:27:28.461528 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:28.461299 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7eee3b-dockercfg-xltvg\"" Oct 30 12:27:28.611322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:28.608786 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pshj\" (UniqueName: \"kubernetes.io/projected/b12e3738-51ee-4043-8f3b-f084779b3b9b-kube-api-access-7pshj\") pod \"payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts\" (UID: \"b12e3738-51ee-4043-8f3b-f084779b3b9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts" Oct 30 12:27:28.611322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:28.608876 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12e3738-51ee-4043-8f3b-f084779b3b9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts\" (UID: \"b12e3738-51ee-4043-8f3b-f084779b3b9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts" Oct 30 12:27:28.611322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:28.609966 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12e3738-51ee-4043-8f3b-f084779b3b9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts\" (UID: \"b12e3738-51ee-4043-8f3b-f084779b3b9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts" Oct 30 12:27:28.683213 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:28.683090 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pshj\" (UniqueName: \"kubernetes.io/projected/b12e3738-51ee-4043-8f3b-f084779b3b9b-kube-api-access-7pshj\") pod \"payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts\" (UID: \"b12e3738-51ee-4043-8f3b-f084779b3b9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts" Oct 30 12:27:28.774881 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:28.774850 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts" Oct 30 12:27:29.086656 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:29.086623 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:29.270454 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:29.270429 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts"] Oct 30 12:27:29.271546 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:27:29.271501 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12e3738_51ee_4043_8f3b_f084779b3b9b.slice/crio-ebb642b3e3e0b24e11d7dfe0e884f3b9fecb0b7d652e98f299e4b531137ebfa2 WatchSource:0}: Error finding container ebb642b3e3e0b24e11d7dfe0e884f3b9fecb0b7d652e98f299e4b531137ebfa2: Status 404 returned error can't find the container with id ebb642b3e3e0b24e11d7dfe0e884f3b9fecb0b7d652e98f299e4b531137ebfa2 Oct 30 12:27:29.772629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:29.772587 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts" event={"ID":"b12e3738-51ee-4043-8f3b-f084779b3b9b","Type":"ContainerStarted","Data":"4c46bfb6d6c0562af27097e66b02c8bb98ce28ec398b788162a01cf9591c9155"} Oct 30 12:27:29.772820 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:29.772639 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts" event={"ID":"b12e3738-51ee-4043-8f3b-f084779b3b9b","Type":"ContainerStarted","Data":"ebb642b3e3e0b24e11d7dfe0e884f3b9fecb0b7d652e98f299e4b531137ebfa2"} Oct 30 12:27:29.772909 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:29.772888 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts" Oct 30 12:27:29.843565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:29.843503 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts" podStartSLOduration=1.843482224 podStartE2EDuration="1.843482224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:27:29.811663016 +0000 UTC m=+2002.921505899" watchObservedRunningTime="2026-10-30 12:27:29.843482224 +0000 UTC m=+2002.953325321" Oct 30 12:27:31.043664 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:31.043541 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:31.162546 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:31.162513 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:31.283104 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:31.283028 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:31.563335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:31.563234 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:33.354034 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:33.354000 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:34.974924 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:34.974889 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": read tcp 10.133.0.2:44440->10.133.0.104:15021: read: connection reset by peer" start-of-body= Oct 30 12:27:34.975360 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:34.974955 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" podUID="2cb74db0-df6e-471c-bce4-7de34b2a33e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": read tcp 10.133.0.2:44440->10.133.0.104:15021: read: connection reset by peer" Oct 30 12:27:35.286450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:35.286344 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:35.699152 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:35.699125 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:35.810136 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:35.810102 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:35.814156 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:27:35.814127 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:27:35.863292 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:35.863230 4413 generic.go:356] "Generic (PLEG): container finished" podID="aa0c1522-725d-43f2-abd8-c37053a1f70c" containerID="de2579e32ebcd73db1abfd137677870b35b605be3cce623e19a42358a1cdabdf" exitCode=0 Oct 30 12:27:35.863423 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:35.863310 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" event={"ID":"aa0c1522-725d-43f2-abd8-c37053a1f70c","Type":"ContainerDied","Data":"de2579e32ebcd73db1abfd137677870b35b605be3cce623e19a42358a1cdabdf"} Oct 30 12:27:35.863765 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:35.863727 4413 scope.go:122] "RemoveContainer" containerID="de2579e32ebcd73db1abfd137677870b35b605be3cce623e19a42358a1cdabdf" Oct 30 12:27:35.865527 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:35.865496 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:27:35.865627 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:35.865587 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:27:35.865627 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:35.865619 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:27:35.866500 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:35.866482 4413 generic.go:356] "Generic (PLEG): container finished" podID="2cb74db0-df6e-471c-bce4-7de34b2a33e3" containerID="9653960bc2c2c48634dec6cb66996a6b53430357553c14e919ec81dbf4a4d690" exitCode=0 Oct 30 12:27:35.866590 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:35.866557 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" event={"ID":"2cb74db0-df6e-471c-bce4-7de34b2a33e3","Type":"ContainerDied","Data":"9653960bc2c2c48634dec6cb66996a6b53430357553c14e919ec81dbf4a4d690"} Oct 30 12:27:35.866921 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:35.866904 4413 scope.go:122] "RemoveContainer" containerID="9653960bc2c2c48634dec6cb66996a6b53430357553c14e919ec81dbf4a4d690" Oct 30 12:27:35.868714 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:35.868688 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:27:35.868789 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:35.868768 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:27:35.868849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:35.868821 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:27:36.875325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:36.875235 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" event={"ID":"2cb74db0-df6e-471c-bce4-7de34b2a33e3","Type":"ContainerStarted","Data":"55f66ad0552618147760f065b20735848b774f655c64da2e36479e18669a6de3"} Oct 30 12:27:36.877323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:36.877294 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" event={"ID":"aa0c1522-725d-43f2-abd8-c37053a1f70c","Type":"ContainerStarted","Data":"f67f129ba3b96e7730f8b6c51a4b5a8c0a33b2c95a354de04f56f7b362f5832f"} Oct 30 12:27:36.955020 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:36.954982 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s"] Oct 30 12:27:37.175339 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:37.172565 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:27:37.175339 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:37.172678 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 12:27:37.175339 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:37.172720 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" podUID="aa0c1522-725d-43f2-abd8-c37053a1f70c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 12:27:37.176994 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:37.176951 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:37.306844 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:37.306812 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:37.359159 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:37.359105 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:27:37.359591 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:37.359553 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 12:27:37.363624 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:37.363577 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" podUID="2cb74db0-df6e-471c-bce4-7de34b2a33e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 12:27:38.129275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:38.128763 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 12:27:38.129275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:38.128825 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" podUID="aa0c1522-725d-43f2-abd8-c37053a1f70c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 12:27:38.350013 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:38.349975 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 12:27:38.350191 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:38.350035 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" podUID="2cb74db0-df6e-471c-bce4-7de34b2a33e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 12:27:39.080840 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:39.080801 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:39.173750 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:39.173712 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 12:27:39.174416 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:39.174373 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" podUID="aa0c1522-725d-43f2-abd8-c37053a1f70c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 12:27:39.350588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:39.350494 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 12:27:39.350588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:39.350554 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" podUID="2cb74db0-df6e-471c-bce4-7de34b2a33e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 12:27:40.135647 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:40.135565 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 12:27:40.135898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:40.135868 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" podUID="aa0c1522-725d-43f2-abd8-c37053a1f70c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 12:27:40.353754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:40.353721 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 12:27:40.354271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:40.354216 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" podUID="2cb74db0-df6e-471c-bce4-7de34b2a33e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 12:27:40.783827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:40.783791 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts" Oct 30 12:27:41.135365 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:41.135332 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 12:27:41.135661 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:41.135632 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" podUID="aa0c1522-725d-43f2-abd8-c37053a1f70c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 12:27:41.350058 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:41.350020 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 12:27:41.350504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:41.350469 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" podUID="2cb74db0-df6e-471c-bce4-7de34b2a33e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 12:27:41.442016 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:41.441894 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:41.955541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:41.955500 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s" podUID="f3e97203-7545-4a2d-9382-3804e526c483" containerName="payload-processing" containerID="cri-o://595c9162c031a33d9e0b3875237f5c7a41b70e82add03aefa767825f830f03b4" gracePeriod=25 Oct 30 12:27:42.128160 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.128129 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 12:27:42.128385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.128192 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" podUID="aa0c1522-725d-43f2-abd8-c37053a1f70c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 12:27:42.273110 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.273079 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s" Oct 30 12:27:42.386396 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.386357 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e97203-7545-4a2d-9382-3804e526c483-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e97203-7545-4a2d-9382-3804e526c483-plugins-config-volume\") pod \"f3e97203-7545-4a2d-9382-3804e526c483\" (UID: \"f3e97203-7545-4a2d-9382-3804e526c483\") " Oct 30 12:27:42.386690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.386666 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e97203-7545-4a2d-9382-3804e526c483-kube-api-access-vtmsh\" (UniqueName: \"kubernetes.io/projected/f3e97203-7545-4a2d-9382-3804e526c483-kube-api-access-vtmsh\") pod \"f3e97203-7545-4a2d-9382-3804e526c483\" (UID: \"f3e97203-7545-4a2d-9382-3804e526c483\") " Oct 30 12:27:42.386842 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.386815 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e97203-7545-4a2d-9382-3804e526c483-plugins-config-volume" pod "f3e97203-7545-4a2d-9382-3804e526c483" (UID: "f3e97203-7545-4a2d-9382-3804e526c483"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:27:42.387348 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.387330 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e97203-7545-4a2d-9382-3804e526c483-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:27:42.389319 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.389285 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e97203-7545-4a2d-9382-3804e526c483-kube-api-access-vtmsh" pod "f3e97203-7545-4a2d-9382-3804e526c483" (UID: "f3e97203-7545-4a2d-9382-3804e526c483"). InnerVolumeSpecName "kube-api-access-vtmsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:42.445734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.445702 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:27:42.446329 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.446300 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:27:42.447171 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.447153 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:27:42.489609 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.489546 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtmsh\" (UniqueName: \"kubernetes.io/projected/f3e97203-7545-4a2d-9382-3804e526c483-kube-api-access-vtmsh\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:27:42.970130 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.970099 4413 generic.go:356] "Generic (PLEG): container finished" podID="f3e97203-7545-4a2d-9382-3804e526c483" containerID="595c9162c031a33d9e0b3875237f5c7a41b70e82add03aefa767825f830f03b4" exitCode=0 Oct 30 12:27:42.970384 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.970150 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s" Oct 30 12:27:42.970384 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.970178 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s" event={"ID":"f3e97203-7545-4a2d-9382-3804e526c483","Type":"ContainerDied","Data":"595c9162c031a33d9e0b3875237f5c7a41b70e82add03aefa767825f830f03b4"} Oct 30 12:27:42.970384 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.970232 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s" event={"ID":"f3e97203-7545-4a2d-9382-3804e526c483","Type":"ContainerDied","Data":"938c919cd232581fd049634d0402878f33245e8e6e864a79d7e4bf8e21917e2f"} Oct 30 12:27:42.970384 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.970281 4413 scope.go:122] "RemoveContainer" containerID="595c9162c031a33d9e0b3875237f5c7a41b70e82add03aefa767825f830f03b4" Oct 30 12:27:42.981133 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.981118 4413 scope.go:122] "RemoveContainer" containerID="595c9162c031a33d9e0b3875237f5c7a41b70e82add03aefa767825f830f03b4" Oct 30 12:27:42.981471 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:27:42.981436 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"595c9162c031a33d9e0b3875237f5c7a41b70e82add03aefa767825f830f03b4\": container with ID starting with 595c9162c031a33d9e0b3875237f5c7a41b70e82add03aefa767825f830f03b4 not found: ID does not exist" containerID="595c9162c031a33d9e0b3875237f5c7a41b70e82add03aefa767825f830f03b4" Oct 30 12:27:42.981553 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:42.981482 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"595c9162c031a33d9e0b3875237f5c7a41b70e82add03aefa767825f830f03b4"} err="failed to get container status \"595c9162c031a33d9e0b3875237f5c7a41b70e82add03aefa767825f830f03b4\": rpc error: code = NotFound desc = could not find container \"595c9162c031a33d9e0b3875237f5c7a41b70e82add03aefa767825f830f03b4\": container with ID starting with 595c9162c031a33d9e0b3875237f5c7a41b70e82add03aefa767825f830f03b4 not found: ID does not exist" Oct 30 12:27:43.025352 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:43.025288 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s"] Oct 30 12:27:43.039958 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:43.039919 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c57af-56d5b8f9c6-fgv7s"] Oct 30 12:27:43.057230 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:43.057189 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:27:43.133732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:43.133702 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:27:43.810009 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:43.809977 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:43.979112 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:43.979087 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:27:45.020181 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:45.020147 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e97203-7545-4a2d-9382-3804e526c483" path="/var/lib/kubelet/pods/f3e97203-7545-4a2d-9382-3804e526c483/volumes" Oct 30 12:27:45.688803 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:45.688769 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:47.634349 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:47.634314 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw"] Oct 30 12:27:47.635122 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:47.635086 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" podUID="fda190d0-d953-4922-a2c7-69d48381a12c" containerName="maas-api" containerID="cri-o://bab0e867e97c07cd3f47546a3abea3474a9f42fa96f29c43167b714d1c79d9b0" gracePeriod=30 Oct 30 12:27:47.694243 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:47.694205 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:47.874953 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:47.874894 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:47.949137 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:47.949115 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" Oct 30 12:27:47.992861 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:47.992824 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:48.009683 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.009646 4413 generic.go:356] "Generic (PLEG): container finished" podID="fda190d0-d953-4922-a2c7-69d48381a12c" containerID="bab0e867e97c07cd3f47546a3abea3474a9f42fa96f29c43167b714d1c79d9b0" exitCode=0 Oct 30 12:27:48.009855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.009729 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" Oct 30 12:27:48.009855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.009744 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" event={"ID":"fda190d0-d953-4922-a2c7-69d48381a12c","Type":"ContainerDied","Data":"bab0e867e97c07cd3f47546a3abea3474a9f42fa96f29c43167b714d1c79d9b0"} Oct 30 12:27:48.009855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.009794 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw" event={"ID":"fda190d0-d953-4922-a2c7-69d48381a12c","Type":"ContainerDied","Data":"fced7da8b8bf58a8d990a728adf55e5dd8684ab6d7e8a41cd4f2213038cb9143"} Oct 30 12:27:48.009855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.009823 4413 scope.go:122] "RemoveContainer" containerID="bab0e867e97c07cd3f47546a3abea3474a9f42fa96f29c43167b714d1c79d9b0" Oct 30 12:27:48.021219 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.021198 4413 scope.go:122] "RemoveContainer" containerID="bab0e867e97c07cd3f47546a3abea3474a9f42fa96f29c43167b714d1c79d9b0" Oct 30 12:27:48.021560 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:27:48.021534 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab0e867e97c07cd3f47546a3abea3474a9f42fa96f29c43167b714d1c79d9b0\": container with ID starting with bab0e867e97c07cd3f47546a3abea3474a9f42fa96f29c43167b714d1c79d9b0 not found: ID does not exist" containerID="bab0e867e97c07cd3f47546a3abea3474a9f42fa96f29c43167b714d1c79d9b0" Oct 30 12:27:48.021630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.021574 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab0e867e97c07cd3f47546a3abea3474a9f42fa96f29c43167b714d1c79d9b0"} err="failed to get container status \"bab0e867e97c07cd3f47546a3abea3474a9f42fa96f29c43167b714d1c79d9b0\": rpc error: code = NotFound desc = could not find container \"bab0e867e97c07cd3f47546a3abea3474a9f42fa96f29c43167b714d1c79d9b0\": container with ID starting with bab0e867e97c07cd3f47546a3abea3474a9f42fa96f29c43167b714d1c79d9b0 not found: ID does not exist" Oct 30 12:27:48.077942 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.077918 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fda190d0-d953-4922-a2c7-69d48381a12c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fda190d0-d953-4922-a2c7-69d48381a12c-maas-api-tls\") pod \"fda190d0-d953-4922-a2c7-69d48381a12c\" (UID: \"fda190d0-d953-4922-a2c7-69d48381a12c\") " Oct 30 12:27:48.078084 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.078010 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fda190d0-d953-4922-a2c7-69d48381a12c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda190d0-d953-4922-a2c7-69d48381a12c-openshift-service-ca\") pod \"fda190d0-d953-4922-a2c7-69d48381a12c\" (UID: \"fda190d0-d953-4922-a2c7-69d48381a12c\") " Oct 30 12:27:48.078152 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.078127 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fda190d0-d953-4922-a2c7-69d48381a12c-kube-api-access-gm77v\" (UniqueName: \"kubernetes.io/projected/fda190d0-d953-4922-a2c7-69d48381a12c-kube-api-access-gm77v\") pod \"fda190d0-d953-4922-a2c7-69d48381a12c\" (UID: \"fda190d0-d953-4922-a2c7-69d48381a12c\") " Oct 30 12:27:48.078208 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.078170 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fda190d0-d953-4922-a2c7-69d48381a12c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fda190d0-d953-4922-a2c7-69d48381a12c-metrics-certs\") pod \"fda190d0-d953-4922-a2c7-69d48381a12c\" (UID: \"fda190d0-d953-4922-a2c7-69d48381a12c\") " Oct 30 12:27:48.078388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.078365 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda190d0-d953-4922-a2c7-69d48381a12c-openshift-service-ca" pod "fda190d0-d953-4922-a2c7-69d48381a12c" (UID: "fda190d0-d953-4922-a2c7-69d48381a12c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:27:48.078925 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.078897 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda190d0-d953-4922-a2c7-69d48381a12c-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.080371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.080351 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fda190d0-d953-4922-a2c7-69d48381a12c-maas-api-tls" pod "fda190d0-d953-4922-a2c7-69d48381a12c" (UID: "fda190d0-d953-4922-a2c7-69d48381a12c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:27:48.080482 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.080463 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fda190d0-d953-4922-a2c7-69d48381a12c-metrics-certs" pod "fda190d0-d953-4922-a2c7-69d48381a12c" (UID: "fda190d0-d953-4922-a2c7-69d48381a12c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:27:48.080531 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.080482 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda190d0-d953-4922-a2c7-69d48381a12c-kube-api-access-gm77v" pod "fda190d0-d953-4922-a2c7-69d48381a12c" (UID: "fda190d0-d953-4922-a2c7-69d48381a12c"). InnerVolumeSpecName "kube-api-access-gm77v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:48.182418 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.181527 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm77v\" (UniqueName: \"kubernetes.io/projected/fda190d0-d953-4922-a2c7-69d48381a12c-kube-api-access-gm77v\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.182418 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.181561 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fda190d0-d953-4922-a2c7-69d48381a12c-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.182418 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.181576 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fda190d0-d953-4922-a2c7-69d48381a12c-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:27:48.291731 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.291689 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:48.358394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.358352 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw"] Oct 30 12:27:48.371987 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.371949 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-951ad1-7b9ddd8f6b-bpttw"] Oct 30 12:27:48.611170 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.611057 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:48.731674 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:48.731643 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:49.020104 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:49.020026 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fda190d0-d953-4922-a2c7-69d48381a12c" path="/var/lib/kubelet/pods/fda190d0-d953-4922-a2c7-69d48381a12c/volumes" Oct 30 12:27:49.066074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:49.066050 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:50.254890 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:50.254846 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:52.016284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:52.016235 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:53.255984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:53.255940 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:53.432047 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:53.432011 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:54.327445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.327416 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh"] Oct 30 12:27:54.328089 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.328068 4413 cpu_manager.go:414] "containerMap: removing container" podUID="fda190d0-d953-4922-a2c7-69d48381a12c" containerName="maas-api" Oct 30 12:27:54.328089 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.328087 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda190d0-d953-4922-a2c7-69d48381a12c" containerName="maas-api" Oct 30 12:27:54.328263 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.328113 4413 cpu_manager.go:414] "containerMap: removing container" podUID="f3e97203-7545-4a2d-9382-3804e526c483" containerName="payload-processing" Oct 30 12:27:54.328263 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.328119 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e97203-7545-4a2d-9382-3804e526c483" containerName="payload-processing" Oct 30 12:27:54.328263 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.328200 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda190d0-d953-4922-a2c7-69d48381a12c" containerName="maas-api" Oct 30 12:27:54.328263 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.328211 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e97203-7545-4a2d-9382-3804e526c483" containerName="payload-processing" Oct 30 12:27:54.331566 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.331546 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:27:54.349034 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.349008 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5c29\" (UniqueName: \"kubernetes.io/projected/a1a91278-7714-4ff4-a35a-5f573a09497a-kube-api-access-w5c29\") pod \"maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh\" (UID: \"a1a91278-7714-4ff4-a35a-5f573a09497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:27:54.349512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.349224 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a91278-7714-4ff4-a35a-5f573a09497a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh\" (UID: \"a1a91278-7714-4ff4-a35a-5f573a09497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:27:54.349512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.349370 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a91278-7714-4ff4-a35a-5f573a09497a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh\" (UID: \"a1a91278-7714-4ff4-a35a-5f573a09497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:27:54.349747 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.349714 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a91278-7714-4ff4-a35a-5f573a09497a-metrics-certs\") pod \"maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh\" (UID: \"a1a91278-7714-4ff4-a35a-5f573a09497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:27:54.354236 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.354214 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh"] Oct 30 12:27:54.360315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.360291 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7eee3b-serving-cert\"" Oct 30 12:27:54.428298 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.428264 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:54.451144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.451113 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a91278-7714-4ff4-a35a-5f573a09497a-metrics-certs\") pod \"maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh\" (UID: \"a1a91278-7714-4ff4-a35a-5f573a09497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:27:54.451341 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.451286 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5c29\" (UniqueName: \"kubernetes.io/projected/a1a91278-7714-4ff4-a35a-5f573a09497a-kube-api-access-w5c29\") pod \"maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh\" (UID: \"a1a91278-7714-4ff4-a35a-5f573a09497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:27:54.451341 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.451305 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a91278-7714-4ff4-a35a-5f573a09497a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh\" (UID: \"a1a91278-7714-4ff4-a35a-5f573a09497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:27:54.451435 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.451355 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a91278-7714-4ff4-a35a-5f573a09497a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh\" (UID: \"a1a91278-7714-4ff4-a35a-5f573a09497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:27:54.451529 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:27:54.451512 4413 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-serving-cert: secret "maas-api-e2e-ipp-a-7eee3b-serving-cert" not found Oct 30 12:27:54.451613 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:27:54.451604 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1a91278-7714-4ff4-a35a-5f573a09497a-maas-api-tls podName:a1a91278-7714-4ff4-a35a-5f573a09497a nodeName:}" failed. No retries permitted until 2026-10-30 12:27:54.951579107 +0000 UTC m=+2028.061421967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a1a91278-7714-4ff4-a35a-5f573a09497a-maas-api-tls") pod "maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" (UID: "a1a91278-7714-4ff4-a35a-5f573a09497a") : secret "maas-api-e2e-ipp-a-7eee3b-serving-cert" not found Oct 30 12:27:54.451957 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.451938 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a91278-7714-4ff4-a35a-5f573a09497a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh\" (UID: \"a1a91278-7714-4ff4-a35a-5f573a09497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:27:54.453684 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.453665 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a91278-7714-4ff4-a35a-5f573a09497a-metrics-certs\") pod \"maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh\" (UID: \"a1a91278-7714-4ff4-a35a-5f573a09497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:27:54.477045 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.477023 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5c29\" (UniqueName: \"kubernetes.io/projected/a1a91278-7714-4ff4-a35a-5f573a09497a-kube-api-access-w5c29\") pod \"maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh\" (UID: \"a1a91278-7714-4ff4-a35a-5f573a09497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:27:54.549786 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.549756 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:54.804848 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.804813 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:54.958342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.958303 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a91278-7714-4ff4-a35a-5f573a09497a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh\" (UID: \"a1a91278-7714-4ff4-a35a-5f573a09497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:27:54.960933 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:54.960906 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a91278-7714-4ff4-a35a-5f573a09497a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh\" (UID: \"a1a91278-7714-4ff4-a35a-5f573a09497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:27:55.246602 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:55.246571 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:27:55.582100 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:55.582056 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh"] Oct 30 12:27:55.593541 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:27:55.593428 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a91278_7714_4ff4_a35a_5f573a09497a.slice/crio-58243f44433c1e7edfdf0c93d9edb572fdbf9588b1705f3bedf1374be03bfab6 WatchSource:0}: Error finding container 58243f44433c1e7edfdf0c93d9edb572fdbf9588b1705f3bedf1374be03bfab6: Status 404 returned error can't find the container with id 58243f44433c1e7edfdf0c93d9edb572fdbf9588b1705f3bedf1374be03bfab6 Oct 30 12:27:55.790774 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:55.790732 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:55.901548 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:55.901514 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:56.065493 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:56.065462 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" event={"ID":"a1a91278-7714-4ff4-a35a-5f573a09497a","Type":"ContainerStarted","Data":"9c0aa062c5521dfb1fd86bd139f16ce992339d622d3de0134fcf9527d290f1d0"} Oct 30 12:27:56.065651 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:56.065517 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" event={"ID":"a1a91278-7714-4ff4-a35a-5f573a09497a","Type":"ContainerStarted","Data":"58243f44433c1e7edfdf0c93d9edb572fdbf9588b1705f3bedf1374be03bfab6"} Oct 30 12:27:56.065651 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:56.065562 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:27:56.188092 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:56.187985 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" podStartSLOduration=1.80994133 podStartE2EDuration="2.187967275s" totalImagesPullingTime="378.025945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:27:56.18591805 +0000 UTC m=+2029.295760933" watchObservedRunningTime="2026-10-30 12:27:56.187967275 +0000 UTC m=+2029.297810156" Oct 30 12:27:56.266403 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:56.266370 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f"] Oct 30 12:27:56.266641 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:56.266617 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" podUID="30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" containerName="istio-proxy" containerID="cri-o://aef235c22e3ec8d05b0db311520dcf2728817a67fc0981c3672d3786dfd41886" gracePeriod=30 Oct 30 12:27:56.287565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:56.287534 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:56.462126 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:56.462026 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:57.673904 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:57.673866 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:58.432634 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:58.432595 4413 patch_prober.go:29] interesting pod/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:27:58.432811 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:58.432652 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" podUID="30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:27:58.660340 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:58.660298 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:58.788840 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:58.788752 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk"] Oct 30 12:27:58.789356 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:58.789103 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" podUID="c0ee5a0e-d8d9-4823-9ebd-512e246c744b" containerName="payload-pre-processing" containerID="cri-o://d7763c84e920fbf347e2233c04eceda565bc2fd883a184788f568ead10df20d5" gracePeriod=30 Oct 30 12:27:58.848947 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:58.848911 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:58.905975 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:58.905936 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" podUID="c0ee5a0e-d8d9-4823-9ebd-512e246c744b" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.100:9004: connect: connection refused" Oct 30 12:27:58.973680 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:58.973544 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:59.202031 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:59.201896 4413 generic.go:356] "Generic (PLEG): container finished" podID="c0ee5a0e-d8d9-4823-9ebd-512e246c744b" containerID="d7763c84e920fbf347e2233c04eceda565bc2fd883a184788f568ead10df20d5" exitCode=0 Oct 30 12:27:59.202314 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:59.202090 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" event={"ID":"c0ee5a0e-d8d9-4823-9ebd-512e246c744b","Type":"ContainerDied","Data":"d7763c84e920fbf347e2233c04eceda565bc2fd883a184788f568ead10df20d5"} Oct 30 12:27:59.202314 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:59.202162 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" event={"ID":"c0ee5a0e-d8d9-4823-9ebd-512e246c744b","Type":"ContainerDied","Data":"a2e55bd291f7528d30d8e494bd472332446fc34c369d2fa97895f5e1a8fb4836"} Oct 30 12:27:59.202783 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:59.202489 4413 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2e55bd291f7528d30d8e494bd472332446fc34c369d2fa97895f5e1a8fb4836" Oct 30 12:27:59.203804 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:59.203491 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" Oct 30 12:27:59.237721 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:59.237677 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:59.243448 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:59.243423 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ee5a0e-d8d9-4823-9ebd-512e246c744b-kube-api-access-gjcnn\" (UniqueName: \"kubernetes.io/projected/c0ee5a0e-d8d9-4823-9ebd-512e246c744b-kube-api-access-gjcnn\") pod \"c0ee5a0e-d8d9-4823-9ebd-512e246c744b\" (UID: \"c0ee5a0e-d8d9-4823-9ebd-512e246c744b\") " Oct 30 12:27:59.243586 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:59.243568 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ee5a0e-d8d9-4823-9ebd-512e246c744b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ee5a0e-d8d9-4823-9ebd-512e246c744b-plugins-config-volume\") pod \"c0ee5a0e-d8d9-4823-9ebd-512e246c744b\" (UID: \"c0ee5a0e-d8d9-4823-9ebd-512e246c744b\") " Oct 30 12:27:59.243910 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:59.243888 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ee5a0e-d8d9-4823-9ebd-512e246c744b-plugins-config-volume" pod "c0ee5a0e-d8d9-4823-9ebd-512e246c744b" (UID: "c0ee5a0e-d8d9-4823-9ebd-512e246c744b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:27:59.244371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:59.244345 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ee5a0e-d8d9-4823-9ebd-512e246c744b-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:27:59.245864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:59.245840 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ee5a0e-d8d9-4823-9ebd-512e246c744b-kube-api-access-gjcnn" pod "c0ee5a0e-d8d9-4823-9ebd-512e246c744b" (UID: "c0ee5a0e-d8d9-4823-9ebd-512e246c744b"). InnerVolumeSpecName "kube-api-access-gjcnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:27:59.346222 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:59.346137 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjcnn\" (UniqueName: \"kubernetes.io/projected/c0ee5a0e-d8d9-4823-9ebd-512e246c744b-kube-api-access-gjcnn\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:27:59.440414 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:59.440384 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:59.550111 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:59.550079 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:27:59.987053 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:27:59.987017 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:00.208168 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:00.208134 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk" Oct 30 12:28:00.289333 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:00.282309 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk"] Oct 30 12:28:00.291752 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:00.291712 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7e2db2-f6d54d7c9-rcfpk"] Oct 30 12:28:01.021348 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.021318 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ee5a0e-d8d9-4823-9ebd-512e246c744b" path="/var/lib/kubelet/pods/c0ee5a0e-d8d9-4823-9ebd-512e246c744b/volumes" Oct 30 12:28:01.493715 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.493672 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:01.563468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.563405 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:28:01.694396 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.694366 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-credential-socket\") pod \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " Oct 30 12:28:01.694396 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.694401 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-envoy\") pod \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " Oct 30 12:28:01.694675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.694455 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-workload-socket\") pod \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " Oct 30 12:28:01.694675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.694630 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-token\" (UniqueName: \"kubernetes.io/projected/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-token\") pod \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " Oct 30 12:28:01.694675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.694661 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-podinfo\") pod \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " Oct 30 12:28:01.694781 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.694691 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-workload-socket" pod "30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" (UID: "30dcb8a5-3b7e-4d7b-8145-996efaf9e84a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:01.694781 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.694753 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-credential-socket" pod "30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" (UID: "30dcb8a5-3b7e-4d7b-8145-996efaf9e84a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:01.694781 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.694766 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-data\") pod \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " Oct 30 12:28:01.694891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.694801 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-workload-certs\") pod \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " Oct 30 12:28:01.694891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.694856 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istiod-ca-cert\") pod \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " Oct 30 12:28:01.694891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.694880 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-kube-api-access-5dd9r\" (UniqueName: \"kubernetes.io/projected/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-kube-api-access-5dd9r\") pod \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\" (UID: \"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a\") " Oct 30 12:28:01.695054 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.695006 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-data" pod "30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" (UID: "30dcb8a5-3b7e-4d7b-8145-996efaf9e84a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:01.695119 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.695048 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-workload-certs" pod "30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" (UID: "30dcb8a5-3b7e-4d7b-8145-996efaf9e84a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:01.695321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.695299 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istiod-ca-cert" pod "30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" (UID: "30dcb8a5-3b7e-4d7b-8145-996efaf9e84a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:01.695771 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.695754 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:01.695834 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.695780 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:01.695834 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.695794 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:01.695834 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.695807 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:01.695834 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.695821 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:01.696968 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.696946 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-podinfo" pod "30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" (UID: "30dcb8a5-3b7e-4d7b-8145-996efaf9e84a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:28:01.697110 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.697093 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-token" pod "30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" (UID: "30dcb8a5-3b7e-4d7b-8145-996efaf9e84a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:01.697206 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.697183 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-envoy" pod "30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" (UID: "30dcb8a5-3b7e-4d7b-8145-996efaf9e84a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:01.697323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.697290 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-kube-api-access-5dd9r" pod "30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" (UID: "30dcb8a5-3b7e-4d7b-8145-996efaf9e84a"). InnerVolumeSpecName "kube-api-access-5dd9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:01.797498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.797462 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:01.797498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.797494 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:01.797498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.797505 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dd9r\" (UniqueName: \"kubernetes.io/projected/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-kube-api-access-5dd9r\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:01.797762 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:01.797516 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:02.077495 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:02.077467 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:28:02.222203 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:02.222171 4413 generic.go:356] "Generic (PLEG): container finished" podID="30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" containerID="aef235c22e3ec8d05b0db311520dcf2728817a67fc0981c3672d3786dfd41886" exitCode=0 Oct 30 12:28:02.222429 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:02.222228 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" Oct 30 12:28:02.222429 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:02.222288 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" event={"ID":"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a","Type":"ContainerDied","Data":"aef235c22e3ec8d05b0db311520dcf2728817a67fc0981c3672d3786dfd41886"} Oct 30 12:28:02.222429 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:02.222349 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f" event={"ID":"30dcb8a5-3b7e-4d7b-8145-996efaf9e84a","Type":"ContainerDied","Data":"450268b80971897d89630c76745e5a5b8721556023c186c730b66abbf3bdf987"} Oct 30 12:28:02.222429 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:02.222373 4413 scope.go:122] "RemoveContainer" containerID="aef235c22e3ec8d05b0db311520dcf2728817a67fc0981c3672d3786dfd41886" Oct 30 12:28:02.241484 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:02.241453 4413 scope.go:122] "RemoveContainer" containerID="fa61216a661768c2ffb1cfc935637ff581c445fce161d3b4f57521d3abf1d12f" Oct 30 12:28:02.251949 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:02.251922 4413 scope.go:122] "RemoveContainer" containerID="aef235c22e3ec8d05b0db311520dcf2728817a67fc0981c3672d3786dfd41886" Oct 30 12:28:02.252327 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:28:02.252308 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef235c22e3ec8d05b0db311520dcf2728817a67fc0981c3672d3786dfd41886\": container with ID starting with aef235c22e3ec8d05b0db311520dcf2728817a67fc0981c3672d3786dfd41886 not found: ID does not exist" containerID="aef235c22e3ec8d05b0db311520dcf2728817a67fc0981c3672d3786dfd41886" Oct 30 12:28:02.252400 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:02.252336 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef235c22e3ec8d05b0db311520dcf2728817a67fc0981c3672d3786dfd41886"} err="failed to get container status \"aef235c22e3ec8d05b0db311520dcf2728817a67fc0981c3672d3786dfd41886\": rpc error: code = NotFound desc = could not find container \"aef235c22e3ec8d05b0db311520dcf2728817a67fc0981c3672d3786dfd41886\": container with ID starting with aef235c22e3ec8d05b0db311520dcf2728817a67fc0981c3672d3786dfd41886 not found: ID does not exist" Oct 30 12:28:02.252400 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:02.252355 4413 scope.go:122] "RemoveContainer" containerID="fa61216a661768c2ffb1cfc935637ff581c445fce161d3b4f57521d3abf1d12f" Oct 30 12:28:02.252646 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:28:02.252625 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa61216a661768c2ffb1cfc935637ff581c445fce161d3b4f57521d3abf1d12f\": container with ID starting with fa61216a661768c2ffb1cfc935637ff581c445fce161d3b4f57521d3abf1d12f not found: ID does not exist" containerID="fa61216a661768c2ffb1cfc935637ff581c445fce161d3b4f57521d3abf1d12f" Oct 30 12:28:02.252690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:02.252656 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa61216a661768c2ffb1cfc935637ff581c445fce161d3b4f57521d3abf1d12f"} err="failed to get container status \"fa61216a661768c2ffb1cfc935637ff581c445fce161d3b4f57521d3abf1d12f\": rpc error: code = NotFound desc = could not find container \"fa61216a661768c2ffb1cfc935637ff581c445fce161d3b4f57521d3abf1d12f\": container with ID starting with fa61216a661768c2ffb1cfc935637ff581c445fce161d3b4f57521d3abf1d12f not found: ID does not exist" Oct 30 12:28:02.276324 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:02.276288 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f"] Oct 30 12:28:02.280686 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:02.280659 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-951ad1-openshift-default-74bd9dcf45-h7q8f"] Oct 30 12:28:02.744861 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:02.744821 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:02.898697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:02.898665 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:03.021397 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:03.021324 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" path="/var/lib/kubelet/pods/30dcb8a5-3b7e-4d7b-8145-996efaf9e84a/volumes" Oct 30 12:28:03.845909 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:03.845873 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:05.577718 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:05.577675 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:05.813940 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:28:05.813906 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:28:06.602177 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:06.602140 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:06.753594 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:06.753556 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:06.928446 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:06.928414 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:07.350790 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:07.350714 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:09.710860 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:09.710823 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:11.944988 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:11.944950 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:13.305831 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:13.305782 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq"] Oct 30 12:28:13.306305 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:13.306064 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" podUID="943846d2-b5db-4c25-94e8-e364fcb506f9" containerName="maas-api" containerID="cri-o://2fe34869c8d88a53635474a13eca22edcddeec7123719891c8d76e8a3ec9d611" gracePeriod=30 Oct 30 12:28:13.374671 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:13.374638 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:13.582098 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:13.582070 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" Oct 30 12:28:13.617900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:13.617868 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:13.693377 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:13.693345 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/943846d2-b5db-4c25-94e8-e364fcb506f9-kube-api-access-v2v89\" (UniqueName: \"kubernetes.io/projected/943846d2-b5db-4c25-94e8-e364fcb506f9-kube-api-access-v2v89\") pod \"943846d2-b5db-4c25-94e8-e364fcb506f9\" (UID: \"943846d2-b5db-4c25-94e8-e364fcb506f9\") " Oct 30 12:28:13.693534 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:13.693471 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/943846d2-b5db-4c25-94e8-e364fcb506f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/943846d2-b5db-4c25-94e8-e364fcb506f9-openshift-service-ca\") pod \"943846d2-b5db-4c25-94e8-e364fcb506f9\" (UID: \"943846d2-b5db-4c25-94e8-e364fcb506f9\") " Oct 30 12:28:13.693577 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:13.693534 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/943846d2-b5db-4c25-94e8-e364fcb506f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/943846d2-b5db-4c25-94e8-e364fcb506f9-metrics-certs\") pod \"943846d2-b5db-4c25-94e8-e364fcb506f9\" (UID: \"943846d2-b5db-4c25-94e8-e364fcb506f9\") " Oct 30 12:28:13.693646 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:13.693632 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/943846d2-b5db-4c25-94e8-e364fcb506f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/943846d2-b5db-4c25-94e8-e364fcb506f9-maas-api-tls\") pod \"943846d2-b5db-4c25-94e8-e364fcb506f9\" (UID: \"943846d2-b5db-4c25-94e8-e364fcb506f9\") " Oct 30 12:28:13.693848 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:13.693826 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943846d2-b5db-4c25-94e8-e364fcb506f9-openshift-service-ca" pod "943846d2-b5db-4c25-94e8-e364fcb506f9" (UID: "943846d2-b5db-4c25-94e8-e364fcb506f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:13.694231 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:13.694157 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/943846d2-b5db-4c25-94e8-e364fcb506f9-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:13.695690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:13.695662 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943846d2-b5db-4c25-94e8-e364fcb506f9-maas-api-tls" pod "943846d2-b5db-4c25-94e8-e364fcb506f9" (UID: "943846d2-b5db-4c25-94e8-e364fcb506f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:28:13.695780 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:13.695740 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943846d2-b5db-4c25-94e8-e364fcb506f9-metrics-certs" pod "943846d2-b5db-4c25-94e8-e364fcb506f9" (UID: "943846d2-b5db-4c25-94e8-e364fcb506f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:28:13.695895 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:13.695878 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943846d2-b5db-4c25-94e8-e364fcb506f9-kube-api-access-v2v89" pod "943846d2-b5db-4c25-94e8-e364fcb506f9" (UID: "943846d2-b5db-4c25-94e8-e364fcb506f9"). InnerVolumeSpecName "kube-api-access-v2v89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:13.744500 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:13.744456 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:13.795118 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:13.795083 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2v89\" (UniqueName: \"kubernetes.io/projected/943846d2-b5db-4c25-94e8-e364fcb506f9-kube-api-access-v2v89\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:13.795118 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:13.795117 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/943846d2-b5db-4c25-94e8-e364fcb506f9-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:13.795118 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:13.795129 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/943846d2-b5db-4c25-94e8-e364fcb506f9-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:14.091623 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:14.091591 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:14.204533 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:14.204498 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:14.323226 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:14.323180 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:14.325730 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:14.325706 4413 generic.go:356] "Generic (PLEG): container finished" podID="943846d2-b5db-4c25-94e8-e364fcb506f9" containerID="2fe34869c8d88a53635474a13eca22edcddeec7123719891c8d76e8a3ec9d611" exitCode=0 Oct 30 12:28:14.325880 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:14.325754 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" event={"ID":"943846d2-b5db-4c25-94e8-e364fcb506f9","Type":"ContainerDied","Data":"2fe34869c8d88a53635474a13eca22edcddeec7123719891c8d76e8a3ec9d611"} Oct 30 12:28:14.325880 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:14.325780 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" Oct 30 12:28:14.325880 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:14.325794 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq" event={"ID":"943846d2-b5db-4c25-94e8-e364fcb506f9","Type":"ContainerDied","Data":"a6902c3b8daa98f63947226f97d21be191f2d24eb15a4b89853a108942f15e58"} Oct 30 12:28:14.325880 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:14.325814 4413 scope.go:122] "RemoveContainer" containerID="2fe34869c8d88a53635474a13eca22edcddeec7123719891c8d76e8a3ec9d611" Oct 30 12:28:14.339064 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:14.339046 4413 scope.go:122] "RemoveContainer" containerID="2fe34869c8d88a53635474a13eca22edcddeec7123719891c8d76e8a3ec9d611" Oct 30 12:28:14.339417 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:28:14.339394 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe34869c8d88a53635474a13eca22edcddeec7123719891c8d76e8a3ec9d611\": container with ID starting with 2fe34869c8d88a53635474a13eca22edcddeec7123719891c8d76e8a3ec9d611 not found: ID does not exist" containerID="2fe34869c8d88a53635474a13eca22edcddeec7123719891c8d76e8a3ec9d611" Oct 30 12:28:14.339510 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:14.339429 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe34869c8d88a53635474a13eca22edcddeec7123719891c8d76e8a3ec9d611"} err="failed to get container status \"2fe34869c8d88a53635474a13eca22edcddeec7123719891c8d76e8a3ec9d611\": rpc error: code = NotFound desc = could not find container \"2fe34869c8d88a53635474a13eca22edcddeec7123719891c8d76e8a3ec9d611\": container with ID starting with 2fe34869c8d88a53635474a13eca22edcddeec7123719891c8d76e8a3ec9d611 not found: ID does not exist" Oct 30 12:28:14.381344 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:14.381314 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq"] Oct 30 12:28:14.388666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:14.388636 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-266486-6fd956cfd6-4kjjq"] Oct 30 12:28:14.735141 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:14.735058 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:15.022353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:15.022276 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="943846d2-b5db-4c25-94e8-e364fcb506f9" path="/var/lib/kubelet/pods/943846d2-b5db-4c25-94e8-e364fcb506f9/volumes" Oct 30 12:28:16.003009 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:16.002977 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:17.845231 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:17.845196 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:19.934512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:19.934482 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:20.730496 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:20.730458 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:20.842684 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:20.842648 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:21.062321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:21.056694 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp"] Oct 30 12:28:21.062321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:21.056962 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" podUID="f1e658f2-c95e-40fc-9602-ac8271892d05" containerName="istio-proxy" containerID="cri-o://6e39776182998c7d2b973c1f0527a13a5d1676cb152a449536abff36457955b5" gracePeriod=30 Oct 30 12:28:21.086195 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:21.086155 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:21.328708 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:21.328622 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:21.515728 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:21.515695 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:21.635164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:21.635133 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:22.161405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:22.161371 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:22.278119 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:22.278083 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:22.395454 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:22.395415 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:22.777947 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:22.777912 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:22.897071 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:22.897047 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:23.290882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:23.290814 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:23.407475 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:23.407434 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:23.982370 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:23.982329 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:28:23.982542 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:23.982434 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" podUID="f1e658f2-c95e-40fc-9602-ac8271892d05" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:28:24.397685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:24.397656 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:24.905885 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:24.905856 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:26.341998 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.341933 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:28:26.400605 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.400573 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e658f2-c95e-40fc-9602-ac8271892d05-kube-api-access-bcgqj\" (UniqueName: \"kubernetes.io/projected/f1e658f2-c95e-40fc-9602-ac8271892d05-kube-api-access-bcgqj\") pod \"f1e658f2-c95e-40fc-9602-ac8271892d05\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " Oct 30 12:28:26.400782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.400658 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-data\") pod \"f1e658f2-c95e-40fc-9602-ac8271892d05\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " Oct 30 12:28:26.400860 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.400812 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-workload-certs\") pod \"f1e658f2-c95e-40fc-9602-ac8271892d05\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " Oct 30 12:28:26.400860 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.400839 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-envoy\") pod \"f1e658f2-c95e-40fc-9602-ac8271892d05\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " Oct 30 12:28:26.400977 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.400859 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1e658f2-c95e-40fc-9602-ac8271892d05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e658f2-c95e-40fc-9602-ac8271892d05-istiod-ca-cert\") pod \"f1e658f2-c95e-40fc-9602-ac8271892d05\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " Oct 30 12:28:26.400977 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.400895 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-workload-socket\") pod \"f1e658f2-c95e-40fc-9602-ac8271892d05\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " Oct 30 12:28:26.400977 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.400919 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-podinfo\") pod \"f1e658f2-c95e-40fc-9602-ac8271892d05\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " Oct 30 12:28:26.401130 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.401050 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-data" pod "f1e658f2-c95e-40fc-9602-ac8271892d05" (UID: "f1e658f2-c95e-40fc-9602-ac8271892d05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:26.401130 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.401057 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-credential-socket\") pod \"f1e658f2-c95e-40fc-9602-ac8271892d05\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " Oct 30 12:28:26.401130 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.401122 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-token\" (UniqueName: \"kubernetes.io/projected/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-token\") pod \"f1e658f2-c95e-40fc-9602-ac8271892d05\" (UID: \"f1e658f2-c95e-40fc-9602-ac8271892d05\") " Oct 30 12:28:26.401326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.401273 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-credential-socket" pod "f1e658f2-c95e-40fc-9602-ac8271892d05" (UID: "f1e658f2-c95e-40fc-9602-ac8271892d05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:26.401862 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.401836 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e658f2-c95e-40fc-9602-ac8271892d05-istiod-ca-cert" pod "f1e658f2-c95e-40fc-9602-ac8271892d05" (UID: "f1e658f2-c95e-40fc-9602-ac8271892d05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:26.402285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.402191 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-workload-socket" pod "f1e658f2-c95e-40fc-9602-ac8271892d05" (UID: "f1e658f2-c95e-40fc-9602-ac8271892d05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:26.402438 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.402409 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-workload-certs" pod "f1e658f2-c95e-40fc-9602-ac8271892d05" (UID: "f1e658f2-c95e-40fc-9602-ac8271892d05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:26.403194 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.403171 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e658f2-c95e-40fc-9602-ac8271892d05-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:26.403363 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.403348 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:26.403511 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.403485 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:26.403626 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.403512 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:26.403626 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.403529 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:26.404608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.404582 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-envoy" pod "f1e658f2-c95e-40fc-9602-ac8271892d05" (UID: "f1e658f2-c95e-40fc-9602-ac8271892d05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:26.404608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.404584 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e658f2-c95e-40fc-9602-ac8271892d05-kube-api-access-bcgqj" pod "f1e658f2-c95e-40fc-9602-ac8271892d05" (UID: "f1e658f2-c95e-40fc-9602-ac8271892d05"). InnerVolumeSpecName "kube-api-access-bcgqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:26.404876 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.404856 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-token" pod "f1e658f2-c95e-40fc-9602-ac8271892d05" (UID: "f1e658f2-c95e-40fc-9602-ac8271892d05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:26.405027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.405007 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-podinfo" pod "f1e658f2-c95e-40fc-9602-ac8271892d05" (UID: "f1e658f2-c95e-40fc-9602-ac8271892d05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:28:26.427344 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.427304 4413 generic.go:356] "Generic (PLEG): container finished" podID="f1e658f2-c95e-40fc-9602-ac8271892d05" containerID="6e39776182998c7d2b973c1f0527a13a5d1676cb152a449536abff36457955b5" exitCode=0 Oct 30 12:28:26.427470 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.427360 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" event={"ID":"f1e658f2-c95e-40fc-9602-ac8271892d05","Type":"ContainerDied","Data":"6e39776182998c7d2b973c1f0527a13a5d1676cb152a449536abff36457955b5"} Oct 30 12:28:26.427470 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.427368 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" Oct 30 12:28:26.427470 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.427406 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp" event={"ID":"f1e658f2-c95e-40fc-9602-ac8271892d05","Type":"ContainerDied","Data":"793c542ee6d76135645fa678685f7f8853236536097b6026803cb6037e93631f"} Oct 30 12:28:26.427470 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.427428 4413 scope.go:122] "RemoveContainer" containerID="6e39776182998c7d2b973c1f0527a13a5d1676cb152a449536abff36457955b5" Oct 30 12:28:26.437739 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.437693 4413 scope.go:122] "RemoveContainer" containerID="a6b8bd67a912193cde8efa8f34e6233230f257fdc1b0c337753c74af4bdb575c" Oct 30 12:28:26.447854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.447833 4413 scope.go:122] "RemoveContainer" containerID="6e39776182998c7d2b973c1f0527a13a5d1676cb152a449536abff36457955b5" Oct 30 12:28:26.448152 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:28:26.448134 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e39776182998c7d2b973c1f0527a13a5d1676cb152a449536abff36457955b5\": container with ID starting with 6e39776182998c7d2b973c1f0527a13a5d1676cb152a449536abff36457955b5 not found: ID does not exist" containerID="6e39776182998c7d2b973c1f0527a13a5d1676cb152a449536abff36457955b5" Oct 30 12:28:26.448242 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.448159 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e39776182998c7d2b973c1f0527a13a5d1676cb152a449536abff36457955b5"} err="failed to get container status \"6e39776182998c7d2b973c1f0527a13a5d1676cb152a449536abff36457955b5\": rpc error: code = NotFound desc = could not find container \"6e39776182998c7d2b973c1f0527a13a5d1676cb152a449536abff36457955b5\": container with ID starting with 6e39776182998c7d2b973c1f0527a13a5d1676cb152a449536abff36457955b5 not found: ID does not exist" Oct 30 12:28:26.448242 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.448175 4413 scope.go:122] "RemoveContainer" containerID="a6b8bd67a912193cde8efa8f34e6233230f257fdc1b0c337753c74af4bdb575c" Oct 30 12:28:26.448482 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:28:26.448463 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b8bd67a912193cde8efa8f34e6233230f257fdc1b0c337753c74af4bdb575c\": container with ID starting with a6b8bd67a912193cde8efa8f34e6233230f257fdc1b0c337753c74af4bdb575c not found: ID does not exist" containerID="a6b8bd67a912193cde8efa8f34e6233230f257fdc1b0c337753c74af4bdb575c" Oct 30 12:28:26.448564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.448489 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b8bd67a912193cde8efa8f34e6233230f257fdc1b0c337753c74af4bdb575c"} err="failed to get container status \"a6b8bd67a912193cde8efa8f34e6233230f257fdc1b0c337753c74af4bdb575c\": rpc error: code = NotFound desc = could not find container \"a6b8bd67a912193cde8efa8f34e6233230f257fdc1b0c337753c74af4bdb575c\": container with ID starting with a6b8bd67a912193cde8efa8f34e6233230f257fdc1b0c337753c74af4bdb575c not found: ID does not exist" Oct 30 12:28:26.487541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.487508 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp"] Oct 30 12:28:26.494361 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.494339 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-266486-openshift-default-7b475d5bcc-djlvp"] Oct 30 12:28:26.505394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.505373 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcgqj\" (UniqueName: \"kubernetes.io/projected/f1e658f2-c95e-40fc-9602-ac8271892d05-kube-api-access-bcgqj\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:26.505394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.505395 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:26.505511 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.505405 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:26.505511 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.505413 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1e658f2-c95e-40fc-9602-ac8271892d05-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:26.962417 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:26.962383 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts"] Oct 30 12:28:27.029269 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:27.026443 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1e658f2-c95e-40fc-9602-ac8271892d05" path="/var/lib/kubelet/pods/f1e658f2-c95e-40fc-9602-ac8271892d05/volumes" Oct 30 12:28:27.073987 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:27.073941 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:27.327684 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:27.327563 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:27.449974 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:27.449944 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:27.974549 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:27.974513 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:28.108558 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:28.108514 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:29.216573 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:29.216542 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:29.820838 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:29.820806 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:29.965094 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:29.965050 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:30.266703 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:30.266674 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:30.381650 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:30.381617 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:30.625163 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:30.625137 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv"] Oct 30 12:28:30.625435 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:30.625411 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" podUID="02adbaa5-0ff4-4471-bf4a-d81d9207a485" containerName="maas-api" containerID="cri-o://fee6bf37af89925ea5fb614f8f4007670224280a9f2ff7d2c14518540a76b376" gracePeriod=30 Oct 30 12:28:30.640232 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:30.640197 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:30.754719 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:30.754692 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:30.913878 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:30.913801 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" Oct 30 12:28:31.037733 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.037690 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02adbaa5-0ff4-4471-bf4a-d81d9207a485-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02adbaa5-0ff4-4471-bf4a-d81d9207a485-openshift-service-ca\") pod \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\" (UID: \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\") " Oct 30 12:28:31.037928 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.037802 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02adbaa5-0ff4-4471-bf4a-d81d9207a485-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02adbaa5-0ff4-4471-bf4a-d81d9207a485-maas-api-tls\") pod \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\" (UID: \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\") " Oct 30 12:28:31.037928 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.037906 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02adbaa5-0ff4-4471-bf4a-d81d9207a485-kube-api-access-cw7dn\" (UniqueName: \"kubernetes.io/projected/02adbaa5-0ff4-4471-bf4a-d81d9207a485-kube-api-access-cw7dn\") pod \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\" (UID: \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\") " Oct 30 12:28:31.038054 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.037950 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02adbaa5-0ff4-4471-bf4a-d81d9207a485-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02adbaa5-0ff4-4471-bf4a-d81d9207a485-metrics-certs\") pod \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\" (UID: \"02adbaa5-0ff4-4471-bf4a-d81d9207a485\") " Oct 30 12:28:31.038235 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.038187 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02adbaa5-0ff4-4471-bf4a-d81d9207a485-openshift-service-ca" pod "02adbaa5-0ff4-4471-bf4a-d81d9207a485" (UID: "02adbaa5-0ff4-4471-bf4a-d81d9207a485"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:31.038665 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.038642 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02adbaa5-0ff4-4471-bf4a-d81d9207a485-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:31.044598 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.043410 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02adbaa5-0ff4-4471-bf4a-d81d9207a485-maas-api-tls" pod "02adbaa5-0ff4-4471-bf4a-d81d9207a485" (UID: "02adbaa5-0ff4-4471-bf4a-d81d9207a485"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:28:31.073509 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.070826 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02adbaa5-0ff4-4471-bf4a-d81d9207a485-metrics-certs" pod "02adbaa5-0ff4-4471-bf4a-d81d9207a485" (UID: "02adbaa5-0ff4-4471-bf4a-d81d9207a485"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:28:31.086050 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.085449 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02adbaa5-0ff4-4471-bf4a-d81d9207a485-kube-api-access-cw7dn" pod "02adbaa5-0ff4-4471-bf4a-d81d9207a485" (UID: "02adbaa5-0ff4-4471-bf4a-d81d9207a485"). InnerVolumeSpecName "kube-api-access-cw7dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:31.140412 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.139695 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02adbaa5-0ff4-4471-bf4a-d81d9207a485-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:31.140412 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.139719 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw7dn\" (UniqueName: \"kubernetes.io/projected/02adbaa5-0ff4-4471-bf4a-d81d9207a485-kube-api-access-cw7dn\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:31.140412 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.139728 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02adbaa5-0ff4-4471-bf4a-d81d9207a485-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:31.175748 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.175713 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:31.309067 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.309017 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:31.465437 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.465345 4413 generic.go:356] "Generic (PLEG): container finished" podID="02adbaa5-0ff4-4471-bf4a-d81d9207a485" containerID="fee6bf37af89925ea5fb614f8f4007670224280a9f2ff7d2c14518540a76b376" exitCode=0 Oct 30 12:28:31.465437 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.465403 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" Oct 30 12:28:31.465672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.465528 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" event={"ID":"02adbaa5-0ff4-4471-bf4a-d81d9207a485","Type":"ContainerDied","Data":"fee6bf37af89925ea5fb614f8f4007670224280a9f2ff7d2c14518540a76b376"} Oct 30 12:28:31.465672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.465577 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv" event={"ID":"02adbaa5-0ff4-4471-bf4a-d81d9207a485","Type":"ContainerDied","Data":"c9683621c9493833fd3d9cecdbb1cf75114be17d7d4a9f9a943da2d09b9ed7f8"} Oct 30 12:28:31.465672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.465607 4413 scope.go:122] "RemoveContainer" containerID="fee6bf37af89925ea5fb614f8f4007670224280a9f2ff7d2c14518540a76b376" Oct 30 12:28:31.484995 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.484974 4413 scope.go:122] "RemoveContainer" containerID="fee6bf37af89925ea5fb614f8f4007670224280a9f2ff7d2c14518540a76b376" Oct 30 12:28:31.485531 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:28:31.485502 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee6bf37af89925ea5fb614f8f4007670224280a9f2ff7d2c14518540a76b376\": container with ID starting with fee6bf37af89925ea5fb614f8f4007670224280a9f2ff7d2c14518540a76b376 not found: ID does not exist" containerID="fee6bf37af89925ea5fb614f8f4007670224280a9f2ff7d2c14518540a76b376" Oct 30 12:28:31.485654 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.485543 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee6bf37af89925ea5fb614f8f4007670224280a9f2ff7d2c14518540a76b376"} err="failed to get container status \"fee6bf37af89925ea5fb614f8f4007670224280a9f2ff7d2c14518540a76b376\": rpc error: code = NotFound desc = could not find container \"fee6bf37af89925ea5fb614f8f4007670224280a9f2ff7d2c14518540a76b376\": container with ID starting with fee6bf37af89925ea5fb614f8f4007670224280a9f2ff7d2c14518540a76b376 not found: ID does not exist" Oct 30 12:28:31.519176 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.519145 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv"] Oct 30 12:28:31.524038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.524007 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-431bf6-584fc8c58d-w2pgv"] Oct 30 12:28:31.773268 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.773168 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:31.886695 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.886665 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:31.963187 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:31.963148 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts" podUID="b12e3738-51ee-4043-8f3b-f084779b3b9b" containerName="payload-pre-processing" containerID="cri-o://4c46bfb6d6c0562af27097e66b02c8bb98ce28ec398b788162a01cf9591c9155" gracePeriod=25 Oct 30 12:28:32.012175 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:32.012151 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:32.268741 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:32.268720 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts" Oct 30 12:28:32.355124 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:32.355035 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12e3738-51ee-4043-8f3b-f084779b3b9b-kube-api-access-7pshj\" (UniqueName: \"kubernetes.io/projected/b12e3738-51ee-4043-8f3b-f084779b3b9b-kube-api-access-7pshj\") pod \"b12e3738-51ee-4043-8f3b-f084779b3b9b\" (UID: \"b12e3738-51ee-4043-8f3b-f084779b3b9b\") " Oct 30 12:28:32.355581 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:32.355140 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b12e3738-51ee-4043-8f3b-f084779b3b9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12e3738-51ee-4043-8f3b-f084779b3b9b-plugins-config-volume\") pod \"b12e3738-51ee-4043-8f3b-f084779b3b9b\" (UID: \"b12e3738-51ee-4043-8f3b-f084779b3b9b\") " Oct 30 12:28:32.355783 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:32.355757 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12e3738-51ee-4043-8f3b-f084779b3b9b-plugins-config-volume" pod "b12e3738-51ee-4043-8f3b-f084779b3b9b" (UID: "b12e3738-51ee-4043-8f3b-f084779b3b9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:32.359540 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:32.359497 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12e3738-51ee-4043-8f3b-f084779b3b9b-kube-api-access-7pshj" pod "b12e3738-51ee-4043-8f3b-f084779b3b9b" (UID: "b12e3738-51ee-4043-8f3b-f084779b3b9b"). InnerVolumeSpecName "kube-api-access-7pshj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:32.456481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:32.456455 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pshj\" (UniqueName: \"kubernetes.io/projected/b12e3738-51ee-4043-8f3b-f084779b3b9b-kube-api-access-7pshj\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:32.456481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:32.456480 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12e3738-51ee-4043-8f3b-f084779b3b9b-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:32.465815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:32.465779 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:32.472828 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:32.472803 4413 generic.go:356] "Generic (PLEG): container finished" podID="b12e3738-51ee-4043-8f3b-f084779b3b9b" containerID="4c46bfb6d6c0562af27097e66b02c8bb98ce28ec398b788162a01cf9591c9155" exitCode=0 Oct 30 12:28:32.472954 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:32.472898 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts" event={"ID":"b12e3738-51ee-4043-8f3b-f084779b3b9b","Type":"ContainerDied","Data":"4c46bfb6d6c0562af27097e66b02c8bb98ce28ec398b788162a01cf9591c9155"} Oct 30 12:28:32.472954 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:32.472931 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts" Oct 30 12:28:32.473054 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:32.472963 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts" event={"ID":"b12e3738-51ee-4043-8f3b-f084779b3b9b","Type":"ContainerDied","Data":"ebb642b3e3e0b24e11d7dfe0e884f3b9fecb0b7d652e98f299e4b531137ebfa2"} Oct 30 12:28:32.473054 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:32.472996 4413 scope.go:122] "RemoveContainer" containerID="4c46bfb6d6c0562af27097e66b02c8bb98ce28ec398b788162a01cf9591c9155" Oct 30 12:28:32.483517 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:32.483497 4413 scope.go:122] "RemoveContainer" containerID="4c46bfb6d6c0562af27097e66b02c8bb98ce28ec398b788162a01cf9591c9155" Oct 30 12:28:32.483854 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:28:32.483832 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c46bfb6d6c0562af27097e66b02c8bb98ce28ec398b788162a01cf9591c9155\": container with ID starting with 4c46bfb6d6c0562af27097e66b02c8bb98ce28ec398b788162a01cf9591c9155 not found: ID does not exist" containerID="4c46bfb6d6c0562af27097e66b02c8bb98ce28ec398b788162a01cf9591c9155" Oct 30 12:28:32.483951 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:32.483868 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c46bfb6d6c0562af27097e66b02c8bb98ce28ec398b788162a01cf9591c9155"} err="failed to get container status \"4c46bfb6d6c0562af27097e66b02c8bb98ce28ec398b788162a01cf9591c9155\": rpc error: code = NotFound desc = could not find container \"4c46bfb6d6c0562af27097e66b02c8bb98ce28ec398b788162a01cf9591c9155\": container with ID starting with 4c46bfb6d6c0562af27097e66b02c8bb98ce28ec398b788162a01cf9591c9155 not found: ID does not exist" Oct 30 12:28:32.526981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:32.526961 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts"] Oct 30 12:28:32.534707 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:32.534687 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-98cf88959-fltts"] Oct 30 12:28:33.021146 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:33.021100 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02adbaa5-0ff4-4471-bf4a-d81d9207a485" path="/var/lib/kubelet/pods/02adbaa5-0ff4-4471-bf4a-d81d9207a485/volumes" Oct 30 12:28:33.021689 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:33.021665 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b12e3738-51ee-4043-8f3b-f084779b3b9b" path="/var/lib/kubelet/pods/b12e3738-51ee-4043-8f3b-f084779b3b9b/volumes" Oct 30 12:28:34.435638 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:34.435601 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:35.814297 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:28:35.814259 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:28:36.083793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:36.083694 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:37.795310 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:37.795277 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:39.334230 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:39.334195 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:39.461029 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:39.460973 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:39.672088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:39.672052 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx"] Oct 30 12:28:39.672438 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:39.672319 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" podUID="5235ef15-0da6-4593-85a6-8d453b249064" containerName="istio-proxy" containerID="cri-o://375ce1a6d4ff55680c5015a61d1cf076bd60bf0d7a300deb3ce7a64c61645190" gracePeriod=30 Oct 30 12:28:39.721509 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:39.721466 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:39.833832 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:39.833799 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:41.362865 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:41.362831 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:41.861526 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:41.861482 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:41.978551 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:41.978518 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:42.560986 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:42.560957 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:42.872610 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:42.872529 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:28:42.872610 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:42.872597 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" podUID="5235ef15-0da6-4593-85a6-8d453b249064" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:28:43.832880 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:43.832850 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:43.975504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:43.975472 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:44.219049 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.219017 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5"] Oct 30 12:28:44.220361 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.220337 4413 cpu_manager.go:414] "containerMap: removing container" podUID="c0ee5a0e-d8d9-4823-9ebd-512e246c744b" containerName="payload-pre-processing" Oct 30 12:28:44.220516 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.220502 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ee5a0e-d8d9-4823-9ebd-512e246c744b" containerName="payload-pre-processing" Oct 30 12:28:44.220669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.220659 4413 cpu_manager.go:414] "containerMap: removing container" podUID="30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" containerName="istio-proxy" Oct 30 12:28:44.220760 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.220750 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" containerName="istio-proxy" Oct 30 12:28:44.220837 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.220829 4413 cpu_manager.go:414] "containerMap: removing container" podUID="30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" containerName="istio-proxy" Oct 30 12:28:44.220914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.220906 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" containerName="istio-proxy" Oct 30 12:28:44.220978 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.220969 4413 cpu_manager.go:414] "containerMap: removing container" podUID="f1e658f2-c95e-40fc-9602-ac8271892d05" containerName="istio-proxy" Oct 30 12:28:44.221041 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.221033 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e658f2-c95e-40fc-9602-ac8271892d05" containerName="istio-proxy" Oct 30 12:28:44.221121 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.221113 4413 cpu_manager.go:414] "containerMap: removing container" podUID="02adbaa5-0ff4-4471-bf4a-d81d9207a485" containerName="maas-api" Oct 30 12:28:44.221180 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.221172 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02adbaa5-0ff4-4471-bf4a-d81d9207a485" containerName="maas-api" Oct 30 12:28:44.221273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.221263 4413 cpu_manager.go:414] "containerMap: removing container" podUID="f1e658f2-c95e-40fc-9602-ac8271892d05" containerName="istio-proxy" Oct 30 12:28:44.221346 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.221337 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e658f2-c95e-40fc-9602-ac8271892d05" containerName="istio-proxy" Oct 30 12:28:44.221415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.221407 4413 cpu_manager.go:414] "containerMap: removing container" podUID="b12e3738-51ee-4043-8f3b-f084779b3b9b" containerName="payload-pre-processing" Oct 30 12:28:44.221474 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.221466 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b12e3738-51ee-4043-8f3b-f084779b3b9b" containerName="payload-pre-processing" Oct 30 12:28:44.221535 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.221528 4413 cpu_manager.go:414] "containerMap: removing container" podUID="943846d2-b5db-4c25-94e8-e364fcb506f9" containerName="maas-api" Oct 30 12:28:44.221608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.221586 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943846d2-b5db-4c25-94e8-e364fcb506f9" containerName="maas-api" Oct 30 12:28:44.221833 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.221820 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ee5a0e-d8d9-4823-9ebd-512e246c744b" containerName="payload-pre-processing" Oct 30 12:28:44.222678 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.222663 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" containerName="istio-proxy" Oct 30 12:28:44.222798 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.222788 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="30dcb8a5-3b7e-4d7b-8145-996efaf9e84a" containerName="istio-proxy" Oct 30 12:28:44.226304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.225763 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e658f2-c95e-40fc-9602-ac8271892d05" containerName="istio-proxy" Oct 30 12:28:44.226304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.225788 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="02adbaa5-0ff4-4471-bf4a-d81d9207a485" containerName="maas-api" Oct 30 12:28:44.226304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.225802 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e658f2-c95e-40fc-9602-ac8271892d05" containerName="istio-proxy" Oct 30 12:28:44.226304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.225812 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="b12e3738-51ee-4043-8f3b-f084779b3b9b" containerName="payload-pre-processing" Oct 30 12:28:44.226304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.225823 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="943846d2-b5db-4c25-94e8-e364fcb506f9" containerName="maas-api" Oct 30 12:28:44.241611 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.241584 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5" Oct 30 12:28:44.247927 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.247901 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5"] Oct 30 12:28:44.269392 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.269370 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7eee3b-dockercfg-klfkm\"" Oct 30 12:28:44.269677 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.269664 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7eee3b\"" Oct 30 12:28:44.343162 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.343123 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8af5dd5-cd57-4892-86dd-54f2b3b203a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5\" (UID: \"d8af5dd5-cd57-4892-86dd-54f2b3b203a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5" Oct 30 12:28:44.343328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.343234 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnlsg\" (UniqueName: \"kubernetes.io/projected/d8af5dd5-cd57-4892-86dd-54f2b3b203a0-kube-api-access-gnlsg\") pod \"payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5\" (UID: \"d8af5dd5-cd57-4892-86dd-54f2b3b203a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5" Oct 30 12:28:44.445387 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.445349 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8af5dd5-cd57-4892-86dd-54f2b3b203a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5\" (UID: \"d8af5dd5-cd57-4892-86dd-54f2b3b203a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5" Oct 30 12:28:44.445587 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.445475 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnlsg\" (UniqueName: \"kubernetes.io/projected/d8af5dd5-cd57-4892-86dd-54f2b3b203a0-kube-api-access-gnlsg\") pod \"payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5\" (UID: \"d8af5dd5-cd57-4892-86dd-54f2b3b203a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5" Oct 30 12:28:44.446131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.446107 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8af5dd5-cd57-4892-86dd-54f2b3b203a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5\" (UID: \"d8af5dd5-cd57-4892-86dd-54f2b3b203a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5" Oct 30 12:28:44.468711 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.468687 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnlsg\" (UniqueName: \"kubernetes.io/projected/d8af5dd5-cd57-4892-86dd-54f2b3b203a0-kube-api-access-gnlsg\") pod \"payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5\" (UID: \"d8af5dd5-cd57-4892-86dd-54f2b3b203a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5" Oct 30 12:28:44.566844 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.566775 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5" Oct 30 12:28:44.870353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.870325 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5"] Oct 30 12:28:44.873772 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:28:44.873746 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8af5dd5_cd57_4892_86dd_54f2b3b203a0.slice/crio-7f61bbd549232c9b86c783f292cd857048c45a58f23269d7a503a68f0bc66387 WatchSource:0}: Error finding container 7f61bbd549232c9b86c783f292cd857048c45a58f23269d7a503a68f0bc66387: Status 404 returned error can't find the container with id 7f61bbd549232c9b86c783f292cd857048c45a58f23269d7a503a68f0bc66387 Oct 30 12:28:44.927177 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.927156 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:28:44.949368 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:44.949339 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:45.054269 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.054223 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-workload-certs\") pod \"5235ef15-0da6-4593-85a6-8d453b249064\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " Oct 30 12:28:45.054555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.054307 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-workload-socket\") pod \"5235ef15-0da6-4593-85a6-8d453b249064\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " Oct 30 12:28:45.054555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.054378 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5235ef15-0da6-4593-85a6-8d453b249064-kube-api-access-c9ft9\" (UniqueName: \"kubernetes.io/projected/5235ef15-0da6-4593-85a6-8d453b249064-kube-api-access-c9ft9\") pod \"5235ef15-0da6-4593-85a6-8d453b249064\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " Oct 30 12:28:45.054555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.054455 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5235ef15-0da6-4593-85a6-8d453b249064-istio-token\" (UniqueName: \"kubernetes.io/projected/5235ef15-0da6-4593-85a6-8d453b249064-istio-token\") pod \"5235ef15-0da6-4593-85a6-8d453b249064\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " Oct 30 12:28:45.054555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.054511 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5235ef15-0da6-4593-85a6-8d453b249064-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5235ef15-0da6-4593-85a6-8d453b249064-istio-podinfo\") pod \"5235ef15-0da6-4593-85a6-8d453b249064\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " Oct 30 12:28:45.054555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.054549 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5235ef15-0da6-4593-85a6-8d453b249064-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5235ef15-0da6-4593-85a6-8d453b249064-istiod-ca-cert\") pod \"5235ef15-0da6-4593-85a6-8d453b249064\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " Oct 30 12:28:45.054832 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.054564 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-workload-certs" pod "5235ef15-0da6-4593-85a6-8d453b249064" (UID: "5235ef15-0da6-4593-85a6-8d453b249064"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:45.054832 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.054615 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-istio-data\") pod \"5235ef15-0da6-4593-85a6-8d453b249064\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " Oct 30 12:28:45.054832 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.054653 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-istio-envoy\") pod \"5235ef15-0da6-4593-85a6-8d453b249064\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " Oct 30 12:28:45.054832 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.054798 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-credential-socket\") pod \"5235ef15-0da6-4593-85a6-8d453b249064\" (UID: \"5235ef15-0da6-4593-85a6-8d453b249064\") " Oct 30 12:28:45.055076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.055053 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-workload-socket" pod "5235ef15-0da6-4593-85a6-8d453b249064" (UID: "5235ef15-0da6-4593-85a6-8d453b249064"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:45.055286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.055268 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-credential-socket" pod "5235ef15-0da6-4593-85a6-8d453b249064" (UID: "5235ef15-0da6-4593-85a6-8d453b249064"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:45.055417 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.055400 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:45.055478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.055423 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:45.055478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.055437 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:45.055755 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.055729 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5235ef15-0da6-4593-85a6-8d453b249064-istiod-ca-cert" pod "5235ef15-0da6-4593-85a6-8d453b249064" (UID: "5235ef15-0da6-4593-85a6-8d453b249064"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:28:45.056582 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.056561 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-istio-data" pod "5235ef15-0da6-4593-85a6-8d453b249064" (UID: "5235ef15-0da6-4593-85a6-8d453b249064"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:45.058018 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.057989 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5235ef15-0da6-4593-85a6-8d453b249064-kube-api-access-c9ft9" pod "5235ef15-0da6-4593-85a6-8d453b249064" (UID: "5235ef15-0da6-4593-85a6-8d453b249064"). InnerVolumeSpecName "kube-api-access-c9ft9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:45.058647 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.058621 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5235ef15-0da6-4593-85a6-8d453b249064-istio-token" pod "5235ef15-0da6-4593-85a6-8d453b249064" (UID: "5235ef15-0da6-4593-85a6-8d453b249064"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:28:45.058849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.058829 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5235ef15-0da6-4593-85a6-8d453b249064-istio-podinfo" pod "5235ef15-0da6-4593-85a6-8d453b249064" (UID: "5235ef15-0da6-4593-85a6-8d453b249064"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:28:45.060131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.060110 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-istio-envoy" pod "5235ef15-0da6-4593-85a6-8d453b249064" (UID: "5235ef15-0da6-4593-85a6-8d453b249064"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:28:45.157114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.157088 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9ft9\" (UniqueName: \"kubernetes.io/projected/5235ef15-0da6-4593-85a6-8d453b249064-kube-api-access-c9ft9\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:45.157114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.157111 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5235ef15-0da6-4593-85a6-8d453b249064-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:45.157114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.157120 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5235ef15-0da6-4593-85a6-8d453b249064-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:45.157344 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.157129 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5235ef15-0da6-4593-85a6-8d453b249064-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:45.157344 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.157136 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:45.157344 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.157144 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5235ef15-0da6-4593-85a6-8d453b249064-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:28:45.557562 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.557536 4413 generic.go:356] "Generic (PLEG): container finished" podID="5235ef15-0da6-4593-85a6-8d453b249064" containerID="375ce1a6d4ff55680c5015a61d1cf076bd60bf0d7a300deb3ce7a64c61645190" exitCode=0 Oct 30 12:28:45.557734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.557593 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" Oct 30 12:28:45.557734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.557617 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" event={"ID":"5235ef15-0da6-4593-85a6-8d453b249064","Type":"ContainerDied","Data":"375ce1a6d4ff55680c5015a61d1cf076bd60bf0d7a300deb3ce7a64c61645190"} Oct 30 12:28:45.557734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.557669 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx" event={"ID":"5235ef15-0da6-4593-85a6-8d453b249064","Type":"ContainerDied","Data":"9a1cb00133a0bbc93f898f38c63aa521c7699dd48f8529073a3b05fcfc9fe21b"} Oct 30 12:28:45.557734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.557692 4413 scope.go:122] "RemoveContainer" containerID="375ce1a6d4ff55680c5015a61d1cf076bd60bf0d7a300deb3ce7a64c61645190" Oct 30 12:28:45.559733 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.559706 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5" event={"ID":"d8af5dd5-cd57-4892-86dd-54f2b3b203a0","Type":"ContainerStarted","Data":"36c2f269f5df316fe2c161d25e3bc1ae45b1b215b3a7539f351effc235feaa48"} Oct 30 12:28:45.559848 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.559759 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5" event={"ID":"d8af5dd5-cd57-4892-86dd-54f2b3b203a0","Type":"ContainerStarted","Data":"7f61bbd549232c9b86c783f292cd857048c45a58f23269d7a503a68f0bc66387"} Oct 30 12:28:45.559848 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.559791 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5" Oct 30 12:28:45.568648 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.568632 4413 scope.go:122] "RemoveContainer" containerID="c4d0454bb4d05d0982eb72d09adb224e1b67f6f1df01d0a62beb452ff10621e3" Oct 30 12:28:45.577608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.577434 4413 scope.go:122] "RemoveContainer" containerID="375ce1a6d4ff55680c5015a61d1cf076bd60bf0d7a300deb3ce7a64c61645190" Oct 30 12:28:45.577750 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:28:45.577732 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"375ce1a6d4ff55680c5015a61d1cf076bd60bf0d7a300deb3ce7a64c61645190\": container with ID starting with 375ce1a6d4ff55680c5015a61d1cf076bd60bf0d7a300deb3ce7a64c61645190 not found: ID does not exist" containerID="375ce1a6d4ff55680c5015a61d1cf076bd60bf0d7a300deb3ce7a64c61645190" Oct 30 12:28:45.577798 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.577760 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375ce1a6d4ff55680c5015a61d1cf076bd60bf0d7a300deb3ce7a64c61645190"} err="failed to get container status \"375ce1a6d4ff55680c5015a61d1cf076bd60bf0d7a300deb3ce7a64c61645190\": rpc error: code = NotFound desc = could not find container \"375ce1a6d4ff55680c5015a61d1cf076bd60bf0d7a300deb3ce7a64c61645190\": container with ID starting with 375ce1a6d4ff55680c5015a61d1cf076bd60bf0d7a300deb3ce7a64c61645190 not found: ID does not exist" Oct 30 12:28:45.577798 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.577775 4413 scope.go:122] "RemoveContainer" containerID="c4d0454bb4d05d0982eb72d09adb224e1b67f6f1df01d0a62beb452ff10621e3" Oct 30 12:28:45.578009 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:28:45.577990 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d0454bb4d05d0982eb72d09adb224e1b67f6f1df01d0a62beb452ff10621e3\": container with ID starting with c4d0454bb4d05d0982eb72d09adb224e1b67f6f1df01d0a62beb452ff10621e3 not found: ID does not exist" containerID="c4d0454bb4d05d0982eb72d09adb224e1b67f6f1df01d0a62beb452ff10621e3" Oct 30 12:28:45.578050 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.578015 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d0454bb4d05d0982eb72d09adb224e1b67f6f1df01d0a62beb452ff10621e3"} err="failed to get container status \"c4d0454bb4d05d0982eb72d09adb224e1b67f6f1df01d0a62beb452ff10621e3\": rpc error: code = NotFound desc = could not find container \"c4d0454bb4d05d0982eb72d09adb224e1b67f6f1df01d0a62beb452ff10621e3\": container with ID starting with c4d0454bb4d05d0982eb72d09adb224e1b67f6f1df01d0a62beb452ff10621e3 not found: ID does not exist" Oct 30 12:28:45.609570 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.609548 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx"] Oct 30 12:28:45.621230 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.621211 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-431bf6-openshift-default-7b68d97df-cpsbx"] Oct 30 12:28:45.652720 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:45.652676 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5" podStartSLOduration=1.652665029 podStartE2EDuration="1.652665029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:45.651802075 +0000 UTC m=+2078.761644956" watchObservedRunningTime="2026-10-30 12:28:45.652665029 +0000 UTC m=+2078.762507912" Oct 30 12:28:46.512340 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:46.512299 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:46.712016 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:46.710228 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:47.019497 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:47.019465 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5235ef15-0da6-4593-85a6-8d453b249064" path="/var/lib/kubelet/pods/5235ef15-0da6-4593-85a6-8d453b249064/volumes" Oct 30 12:28:47.084230 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:47.084049 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:48.167985 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.167957 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:48.282733 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.282706 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:48.340326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.340299 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96"] Oct 30 12:28:48.340856 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.340844 4413 cpu_manager.go:414] "containerMap: removing container" podUID="5235ef15-0da6-4593-85a6-8d453b249064" containerName="istio-proxy" Oct 30 12:28:48.340894 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.340858 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5235ef15-0da6-4593-85a6-8d453b249064" containerName="istio-proxy" Oct 30 12:28:48.340894 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.340877 4413 cpu_manager.go:414] "containerMap: removing container" podUID="5235ef15-0da6-4593-85a6-8d453b249064" containerName="istio-proxy" Oct 30 12:28:48.340972 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.340895 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5235ef15-0da6-4593-85a6-8d453b249064" containerName="istio-proxy" Oct 30 12:28:48.341007 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.340998 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="5235ef15-0da6-4593-85a6-8d453b249064" containerName="istio-proxy" Oct 30 12:28:48.341037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.341008 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="5235ef15-0da6-4593-85a6-8d453b249064" containerName="istio-proxy" Oct 30 12:28:48.345776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.345758 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96" Oct 30 12:28:48.363722 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.363693 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96"] Oct 30 12:28:48.371027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.371005 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-41bdda-dockercfg-6q4v8\"" Oct 30 12:28:48.371027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.371005 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-41bdda\"" Oct 30 12:28:48.401198 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.401155 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lb8s\" (UniqueName: \"kubernetes.io/projected/719def6b-4f47-4d2d-8665-a7fa3af2311f-kube-api-access-5lb8s\") pod \"payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96\" (UID: \"719def6b-4f47-4d2d-8665-a7fa3af2311f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96" Oct 30 12:28:48.401488 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.401468 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719def6b-4f47-4d2d-8665-a7fa3af2311f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96\" (UID: \"719def6b-4f47-4d2d-8665-a7fa3af2311f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96" Oct 30 12:28:48.425536 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.425464 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h"] Oct 30 12:28:48.429734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.429714 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h" Oct 30 12:28:48.467188 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.467154 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h"] Oct 30 12:28:48.504718 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.504680 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lb8s\" (UniqueName: \"kubernetes.io/projected/719def6b-4f47-4d2d-8665-a7fa3af2311f-kube-api-access-5lb8s\") pod \"payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96\" (UID: \"719def6b-4f47-4d2d-8665-a7fa3af2311f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96" Oct 30 12:28:48.506545 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.506514 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ad34b9-579c-489a-bef6-f595989c2e86-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h\" (UID: \"56ad34b9-579c-489a-bef6-f595989c2e86\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h" Oct 30 12:28:48.506906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.506879 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffw8j\" (UniqueName: \"kubernetes.io/projected/56ad34b9-579c-489a-bef6-f595989c2e86-kube-api-access-ffw8j\") pod \"payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h\" (UID: \"56ad34b9-579c-489a-bef6-f595989c2e86\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h" Oct 30 12:28:48.507194 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.507166 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719def6b-4f47-4d2d-8665-a7fa3af2311f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96\" (UID: \"719def6b-4f47-4d2d-8665-a7fa3af2311f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96" Oct 30 12:28:48.508286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.508189 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719def6b-4f47-4d2d-8665-a7fa3af2311f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96\" (UID: \"719def6b-4f47-4d2d-8665-a7fa3af2311f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96" Oct 30 12:28:48.528549 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.528525 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lb8s\" (UniqueName: \"kubernetes.io/projected/719def6b-4f47-4d2d-8665-a7fa3af2311f-kube-api-access-5lb8s\") pod \"payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96\" (UID: \"719def6b-4f47-4d2d-8665-a7fa3af2311f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96" Oct 30 12:28:48.608264 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.608219 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ad34b9-579c-489a-bef6-f595989c2e86-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h\" (UID: \"56ad34b9-579c-489a-bef6-f595989c2e86\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h" Oct 30 12:28:48.608437 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.608414 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffw8j\" (UniqueName: \"kubernetes.io/projected/56ad34b9-579c-489a-bef6-f595989c2e86-kube-api-access-ffw8j\") pod \"payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h\" (UID: \"56ad34b9-579c-489a-bef6-f595989c2e86\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h" Oct 30 12:28:48.608988 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.608971 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ad34b9-579c-489a-bef6-f595989c2e86-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h\" (UID: \"56ad34b9-579c-489a-bef6-f595989c2e86\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h" Oct 30 12:28:48.631898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.631875 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffw8j\" (UniqueName: \"kubernetes.io/projected/56ad34b9-579c-489a-bef6-f595989c2e86-kube-api-access-ffw8j\") pod \"payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h\" (UID: \"56ad34b9-579c-489a-bef6-f595989c2e86\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h" Oct 30 12:28:48.658978 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.658958 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96" Oct 30 12:28:48.786231 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.786190 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h" Oct 30 12:28:48.961394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:48.961357 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96"] Oct 30 12:28:48.962708 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:28:48.962654 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719def6b_4f47_4d2d_8665_a7fa3af2311f.slice/crio-30a5c2e546bac017d8560e2019102748bc4a2ab1aee29fb1ab036f4504f913b6 WatchSource:0}: Error finding container 30a5c2e546bac017d8560e2019102748bc4a2ab1aee29fb1ab036f4504f913b6: Status 404 returned error can't find the container with id 30a5c2e546bac017d8560e2019102748bc4a2ab1aee29fb1ab036f4504f913b6 Oct 30 12:28:49.120087 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:49.120051 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h"] Oct 30 12:28:49.120503 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:28:49.120480 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ad34b9_579c_489a_bef6_f595989c2e86.slice/crio-0e9bd708a4493007c174d42933be7d0cd05845ab69afbe8bdb31dd92f1320755 WatchSource:0}: Error finding container 0e9bd708a4493007c174d42933be7d0cd05845ab69afbe8bdb31dd92f1320755: Status 404 returned error can't find the container with id 0e9bd708a4493007c174d42933be7d0cd05845ab69afbe8bdb31dd92f1320755 Oct 30 12:28:49.439064 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:49.439035 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:49.559611 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:49.559566 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:49.592998 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:49.592947 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h" event={"ID":"56ad34b9-579c-489a-bef6-f595989c2e86","Type":"ContainerStarted","Data":"1d80f1b23f02fb8e4751d51910c134962a0d1998e6f922f1a756165ab95b2a83"} Oct 30 12:28:49.593228 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:49.593025 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h" event={"ID":"56ad34b9-579c-489a-bef6-f595989c2e86","Type":"ContainerStarted","Data":"0e9bd708a4493007c174d42933be7d0cd05845ab69afbe8bdb31dd92f1320755"} Oct 30 12:28:49.593478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:49.593456 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h" Oct 30 12:28:49.596399 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:49.596371 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96" event={"ID":"719def6b-4f47-4d2d-8665-a7fa3af2311f","Type":"ContainerStarted","Data":"2f008856d0d32d8dd34f4737cf68642d203eaab83ec4810e42d61bfdd12b0ae7"} Oct 30 12:28:49.596399 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:49.596420 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96" event={"ID":"719def6b-4f47-4d2d-8665-a7fa3af2311f","Type":"ContainerStarted","Data":"30a5c2e546bac017d8560e2019102748bc4a2ab1aee29fb1ab036f4504f913b6"} Oct 30 12:28:49.596697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:49.596669 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96" Oct 30 12:28:49.635331 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:49.635278 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h" podStartSLOduration=1.6352626460000002 podStartE2EDuration="1.635262646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:49.633366421 +0000 UTC m=+2082.743209304" watchObservedRunningTime="2026-10-30 12:28:49.635262646 +0000 UTC m=+2082.745105521" Oct 30 12:28:49.673215 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:49.673167 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96" podStartSLOduration=1.6731536249999999 podStartE2EDuration="1.673153625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:49.672579969 +0000 UTC m=+2082.782422853" watchObservedRunningTime="2026-10-30 12:28:49.673153625 +0000 UTC m=+2082.782996506" Oct 30 12:28:49.682619 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:49.682562 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:51.149619 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:51.149584 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:52.690728 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:52.690685 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:54.377076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:54.377040 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:56.523587 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:56.523554 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:56.570035 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:56.570008 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5" Oct 30 12:28:57.803515 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:57.803484 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:58.083641 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:58.083551 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:59.230081 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.230045 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:59.682153 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.682095 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:59.690826 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.690790 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj"] Oct 30 12:28:59.697372 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.697346 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.706296 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.705925 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj"] Oct 30 12:28:59.712946 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.712913 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-7f78d4-openshift-default-dockercfg-79xjw\"" Oct 30 12:28:59.755647 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.755610 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7kq4\" (UniqueName: \"kubernetes.io/projected/0250c9a9-b527-47e2-8d57-f221fcbd8e19-kube-api-access-m7kq4\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.755848 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.755738 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-workload-socket\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.755848 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.755840 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-envoy\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.755986 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.755962 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istiod-ca-cert\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.756048 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.756004 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-podinfo\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.756138 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.756123 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-token\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.756196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.756150 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-data\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.756245 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.756207 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-workload-certs\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.756245 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.756273 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-credential-socket\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.807284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.805066 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:28:59.857995 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.857961 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-envoy\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.858204 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.858024 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istiod-ca-cert\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.858204 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.858053 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-podinfo\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.858204 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.858102 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-token\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.858204 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.858125 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-data\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.858204 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.858178 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-workload-certs\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.858504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.858205 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-credential-socket\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.859288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.858664 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-credential-socket\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.859288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.858702 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-data\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.859288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.858759 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-workload-certs\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.859288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.858758 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7kq4\" (UniqueName: \"kubernetes.io/projected/0250c9a9-b527-47e2-8d57-f221fcbd8e19-kube-api-access-m7kq4\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.859288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.858878 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-workload-socket\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.859288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.859192 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-workload-socket\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.859288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.859209 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istiod-ca-cert\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.861852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.861823 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-envoy\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.861852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.861833 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-podinfo\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.883030 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.883001 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7kq4\" (UniqueName: \"kubernetes.io/projected/0250c9a9-b527-47e2-8d57-f221fcbd8e19-kube-api-access-m7kq4\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:28:59.883354 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:28:59.883334 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-token\") pod \"e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:29:00.022540 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:00.022409 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:29:00.055022 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:00.054916 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:00.347164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:00.347142 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj"] Oct 30 12:29:00.349207 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:29:00.349182 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0250c9a9_b527_47e2_8d57_f221fcbd8e19.slice/crio-a3e44c074cebb6ebb836fb5f43eea0b6fbb8f63f29b4f843b1cbb0cdcef7a47f WatchSource:0}: Error finding container a3e44c074cebb6ebb836fb5f43eea0b6fbb8f63f29b4f843b1cbb0cdcef7a47f: Status 404 returned error can't find the container with id a3e44c074cebb6ebb836fb5f43eea0b6fbb8f63f29b4f843b1cbb0cdcef7a47f Oct 30 12:29:00.351911 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:00.351875 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:29:00.351999 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:00.351944 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:29:00.351999 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:00.351974 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:29:00.606533 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:00.606453 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h" Oct 30 12:29:00.606533 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:00.606514 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96" Oct 30 12:29:00.672999 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:00.672962 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" event={"ID":"0250c9a9-b527-47e2-8d57-f221fcbd8e19","Type":"ContainerStarted","Data":"6f2735db2e680fe4fc391bee550fba4ad193823dbad37128da8fe2bbefc96cb2"} Oct 30 12:29:00.673193 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:00.673031 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" event={"ID":"0250c9a9-b527-47e2-8d57-f221fcbd8e19","Type":"ContainerStarted","Data":"a3e44c074cebb6ebb836fb5f43eea0b6fbb8f63f29b4f843b1cbb0cdcef7a47f"} Oct 30 12:29:00.791389 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:00.791339 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" podStartSLOduration=1.79132403 podStartE2EDuration="1.79132403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:00.788226448 +0000 UTC m=+2093.898069355" watchObservedRunningTime="2026-10-30 12:29:00.79132403 +0000 UTC m=+2093.901166910" Oct 30 12:29:01.023182 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.023149 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:29:01.032557 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.032518 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:29:01.187426 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.187388 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv"] Oct 30 12:29:01.187684 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.187656 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" podUID="cca05f66-33b8-4d90-89d1-b9b7559fcc32" containerName="maas-api" containerID="cri-o://231d5019d84208c748efb31329ce05b61d6a2a760d8bc5488c3c7614fb8a5916" gracePeriod=30 Oct 30 12:29:01.284233 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.284112 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:01.420266 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.420224 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:01.497831 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.497808 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" Oct 30 12:29:01.530470 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.530439 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:01.599410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.599343 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cca05f66-33b8-4d90-89d1-b9b7559fcc32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cca05f66-33b8-4d90-89d1-b9b7559fcc32-maas-api-tls\") pod \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\" (UID: \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\") " Oct 30 12:29:01.599546 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.599448 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cca05f66-33b8-4d90-89d1-b9b7559fcc32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cca05f66-33b8-4d90-89d1-b9b7559fcc32-metrics-certs\") pod \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\" (UID: \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\") " Oct 30 12:29:01.599546 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.599479 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cca05f66-33b8-4d90-89d1-b9b7559fcc32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca05f66-33b8-4d90-89d1-b9b7559fcc32-openshift-service-ca\") pod \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\" (UID: \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\") " Oct 30 12:29:01.599546 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.599519 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca05f66-33b8-4d90-89d1-b9b7559fcc32-kube-api-access-6b927\" (UniqueName: \"kubernetes.io/projected/cca05f66-33b8-4d90-89d1-b9b7559fcc32-kube-api-access-6b927\") pod \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\" (UID: \"cca05f66-33b8-4d90-89d1-b9b7559fcc32\") " Oct 30 12:29:01.599838 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.599814 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca05f66-33b8-4d90-89d1-b9b7559fcc32-openshift-service-ca" pod "cca05f66-33b8-4d90-89d1-b9b7559fcc32" (UID: "cca05f66-33b8-4d90-89d1-b9b7559fcc32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:29:01.599992 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.599976 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca05f66-33b8-4d90-89d1-b9b7559fcc32-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:01.601845 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.601823 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca05f66-33b8-4d90-89d1-b9b7559fcc32-kube-api-access-6b927" pod "cca05f66-33b8-4d90-89d1-b9b7559fcc32" (UID: "cca05f66-33b8-4d90-89d1-b9b7559fcc32"). InnerVolumeSpecName "kube-api-access-6b927". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:01.602313 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.602285 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cca05f66-33b8-4d90-89d1-b9b7559fcc32-maas-api-tls" pod "cca05f66-33b8-4d90-89d1-b9b7559fcc32" (UID: "cca05f66-33b8-4d90-89d1-b9b7559fcc32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:29:01.602313 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.602296 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cca05f66-33b8-4d90-89d1-b9b7559fcc32-metrics-certs" pod "cca05f66-33b8-4d90-89d1-b9b7559fcc32" (UID: "cca05f66-33b8-4d90-89d1-b9b7559fcc32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:29:01.700273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.688147 4413 generic.go:356] "Generic (PLEG): container finished" podID="cca05f66-33b8-4d90-89d1-b9b7559fcc32" containerID="231d5019d84208c748efb31329ce05b61d6a2a760d8bc5488c3c7614fb8a5916" exitCode=0 Oct 30 12:29:01.700273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.689678 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" Oct 30 12:29:01.700273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.700209 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" event={"ID":"cca05f66-33b8-4d90-89d1-b9b7559fcc32","Type":"ContainerDied","Data":"231d5019d84208c748efb31329ce05b61d6a2a760d8bc5488c3c7614fb8a5916"} Oct 30 12:29:01.700583 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.700318 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:29:01.700583 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.700346 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv" event={"ID":"cca05f66-33b8-4d90-89d1-b9b7559fcc32","Type":"ContainerDied","Data":"20a82417903e87968274f66bfb63f5ee32863bd5a4648757bd13d4b07b4721de"} Oct 30 12:29:01.700583 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.700377 4413 scope.go:122] "RemoveContainer" containerID="231d5019d84208c748efb31329ce05b61d6a2a760d8bc5488c3c7614fb8a5916" Oct 30 12:29:01.712122 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.702868 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cca05f66-33b8-4d90-89d1-b9b7559fcc32-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:01.712122 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.702899 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cca05f66-33b8-4d90-89d1-b9b7559fcc32-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:01.712122 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.702912 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b927\" (UniqueName: \"kubernetes.io/projected/cca05f66-33b8-4d90-89d1-b9b7559fcc32-kube-api-access-6b927\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:01.712122 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.703618 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:29:01.761826 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.761792 4413 scope.go:122] "RemoveContainer" containerID="231d5019d84208c748efb31329ce05b61d6a2a760d8bc5488c3c7614fb8a5916" Oct 30 12:29:01.765301 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:29:01.764576 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231d5019d84208c748efb31329ce05b61d6a2a760d8bc5488c3c7614fb8a5916\": container with ID starting with 231d5019d84208c748efb31329ce05b61d6a2a760d8bc5488c3c7614fb8a5916 not found: ID does not exist" containerID="231d5019d84208c748efb31329ce05b61d6a2a760d8bc5488c3c7614fb8a5916" Oct 30 12:29:01.765301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.764610 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231d5019d84208c748efb31329ce05b61d6a2a760d8bc5488c3c7614fb8a5916"} err="failed to get container status \"231d5019d84208c748efb31329ce05b61d6a2a760d8bc5488c3c7614fb8a5916\": rpc error: code = NotFound desc = could not find container \"231d5019d84208c748efb31329ce05b61d6a2a760d8bc5488c3c7614fb8a5916\": container with ID starting with 231d5019d84208c748efb31329ce05b61d6a2a760d8bc5488c3c7614fb8a5916 not found: ID does not exist" Oct 30 12:29:01.811112 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.811070 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv"] Oct 30 12:29:01.816565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.816505 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-096834-5bb64b4f6-4mvpv"] Oct 30 12:29:01.975608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:01.975576 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:02.092007 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:02.091959 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:02.240654 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:02.240519 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:03.020494 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:03.020463 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cca05f66-33b8-4d90-89d1-b9b7559fcc32" path="/var/lib/kubelet/pods/cca05f66-33b8-4d90-89d1-b9b7559fcc32/volumes" Oct 30 12:29:03.124796 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:03.124761 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:03.445471 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:03.445427 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:03.574704 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:03.574672 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:04.715886 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:04.715853 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:05.196154 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.196123 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:05.298046 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.298016 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4"] Oct 30 12:29:05.298688 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.298673 4413 cpu_manager.go:414] "containerMap: removing container" podUID="cca05f66-33b8-4d90-89d1-b9b7559fcc32" containerName="maas-api" Oct 30 12:29:05.298749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.298689 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca05f66-33b8-4d90-89d1-b9b7559fcc32" containerName="maas-api" Oct 30 12:29:05.298801 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.298792 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca05f66-33b8-4d90-89d1-b9b7559fcc32" containerName="maas-api" Oct 30 12:29:05.303575 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.303560 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4" Oct 30 12:29:05.312459 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.312436 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:05.323283 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.323262 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4"] Oct 30 12:29:05.331827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.331740 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7f78d4-dockercfg-dl6sk\"" Oct 30 12:29:05.332208 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.331922 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7f78d4\"" Oct 30 12:29:05.348024 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.348001 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt"] Oct 30 12:29:05.350277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.350226 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4c8p\" (UniqueName: \"kubernetes.io/projected/8543e221-c552-43fc-a7f4-9d21127db4ef-kube-api-access-p4c8p\") pod \"payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4\" (UID: \"8543e221-c552-43fc-a7f4-9d21127db4ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4" Oct 30 12:29:05.350495 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.350477 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8543e221-c552-43fc-a7f4-9d21127db4ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4\" (UID: \"8543e221-c552-43fc-a7f4-9d21127db4ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4" Oct 30 12:29:05.352325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.352307 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt" Oct 30 12:29:05.374825 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.374801 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt"] Oct 30 12:29:05.467353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.463122 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8543e221-c552-43fc-a7f4-9d21127db4ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4\" (UID: \"8543e221-c552-43fc-a7f4-9d21127db4ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4" Oct 30 12:29:05.467353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.463189 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8543e221-c552-43fc-a7f4-9d21127db4ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4\" (UID: \"8543e221-c552-43fc-a7f4-9d21127db4ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4" Oct 30 12:29:05.467353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.463306 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752943e6-f6f7-450b-9e69-0fda21ee5738-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt\" (UID: \"752943e6-f6f7-450b-9e69-0fda21ee5738\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt" Oct 30 12:29:05.467353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.463406 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4c8p\" (UniqueName: \"kubernetes.io/projected/8543e221-c552-43fc-a7f4-9d21127db4ef-kube-api-access-p4c8p\") pod \"payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4\" (UID: \"8543e221-c552-43fc-a7f4-9d21127db4ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4" Oct 30 12:29:05.467353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.463447 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zms6\" (UniqueName: \"kubernetes.io/projected/752943e6-f6f7-450b-9e69-0fda21ee5738-kube-api-access-7zms6\") pod \"payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt\" (UID: \"752943e6-f6f7-450b-9e69-0fda21ee5738\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt" Oct 30 12:29:05.492383 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.492355 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4c8p\" (UniqueName: \"kubernetes.io/projected/8543e221-c552-43fc-a7f4-9d21127db4ef-kube-api-access-p4c8p\") pod \"payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4\" (UID: \"8543e221-c552-43fc-a7f4-9d21127db4ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4" Oct 30 12:29:05.564932 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.564902 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752943e6-f6f7-450b-9e69-0fda21ee5738-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt\" (UID: \"752943e6-f6f7-450b-9e69-0fda21ee5738\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt" Oct 30 12:29:05.565106 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.564980 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zms6\" (UniqueName: \"kubernetes.io/projected/752943e6-f6f7-450b-9e69-0fda21ee5738-kube-api-access-7zms6\") pod \"payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt\" (UID: \"752943e6-f6f7-450b-9e69-0fda21ee5738\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt" Oct 30 12:29:05.565781 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.565757 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752943e6-f6f7-450b-9e69-0fda21ee5738-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt\" (UID: \"752943e6-f6f7-450b-9e69-0fda21ee5738\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt" Oct 30 12:29:05.613910 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.613887 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zms6\" (UniqueName: \"kubernetes.io/projected/752943e6-f6f7-450b-9e69-0fda21ee5738-kube-api-access-7zms6\") pod \"payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt\" (UID: \"752943e6-f6f7-450b-9e69-0fda21ee5738\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt" Oct 30 12:29:05.617787 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.617768 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4" Oct 30 12:29:05.670058 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.670022 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt" Oct 30 12:29:05.816201 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:29:05.816166 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:29:05.922143 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:05.922118 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4"] Oct 30 12:29:05.922871 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:29:05.922845 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8543e221_c552_43fc_a7f4_9d21127db4ef.slice/crio-74f95502d7807f5fd3de861f04a0c77abced25b3f9157a14a3e6c490538f24ac WatchSource:0}: Error finding container 74f95502d7807f5fd3de861f04a0c77abced25b3f9157a14a3e6c490538f24ac: Status 404 returned error can't find the container with id 74f95502d7807f5fd3de861f04a0c77abced25b3f9157a14a3e6c490538f24ac Oct 30 12:29:06.000672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.000645 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt"] Oct 30 12:29:06.001647 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:29:06.001624 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752943e6_f6f7_450b_9e69_0fda21ee5738.slice/crio-223d24079b2aea324d31ff279b173a00d006fb9e30d7dd69ee87baf110663d72 WatchSource:0}: Error finding container 223d24079b2aea324d31ff279b173a00d006fb9e30d7dd69ee87baf110663d72: Status 404 returned error can't find the container with id 223d24079b2aea324d31ff279b173a00d006fb9e30d7dd69ee87baf110663d72 Oct 30 12:29:06.526775 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.526737 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72"] Oct 30 12:29:06.534317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.534285 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:29:06.548840 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.548812 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72"] Oct 30 12:29:06.562939 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.562549 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-41bdda-serving-cert\"" Oct 30 12:29:06.577569 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.577538 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6159985a-714a-4a43-8645-9865f379b1a3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72\" (UID: \"6159985a-714a-4a43-8645-9865f379b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:29:06.577759 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.577739 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6159985a-714a-4a43-8645-9865f379b1a3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72\" (UID: \"6159985a-714a-4a43-8645-9865f379b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:29:06.577841 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.577803 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb7vp\" (UniqueName: \"kubernetes.io/projected/6159985a-714a-4a43-8645-9865f379b1a3-kube-api-access-qb7vp\") pod \"maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72\" (UID: \"6159985a-714a-4a43-8645-9865f379b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:29:06.577908 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.577896 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6159985a-714a-4a43-8645-9865f379b1a3-metrics-certs\") pod \"maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72\" (UID: \"6159985a-714a-4a43-8645-9865f379b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:29:06.626731 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.626692 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:06.679632 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.679605 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6159985a-714a-4a43-8645-9865f379b1a3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72\" (UID: \"6159985a-714a-4a43-8645-9865f379b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:29:06.679821 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.679663 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb7vp\" (UniqueName: \"kubernetes.io/projected/6159985a-714a-4a43-8645-9865f379b1a3-kube-api-access-qb7vp\") pod \"maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72\" (UID: \"6159985a-714a-4a43-8645-9865f379b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:29:06.679821 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.679785 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6159985a-714a-4a43-8645-9865f379b1a3-metrics-certs\") pod \"maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72\" (UID: \"6159985a-714a-4a43-8645-9865f379b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:29:06.679903 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.679848 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6159985a-714a-4a43-8645-9865f379b1a3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72\" (UID: \"6159985a-714a-4a43-8645-9865f379b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:29:06.680034 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:29:06.680015 4413 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-serving-cert: secret "maas-api-e2e-praxis-a-41bdda-serving-cert" not found Oct 30 12:29:06.680114 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:29:06.680103 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6159985a-714a-4a43-8645-9865f379b1a3-maas-api-tls podName:6159985a-714a-4a43-8645-9865f379b1a3 nodeName:}" failed. No retries permitted until 2026-10-30 12:29:07.180079926 +0000 UTC m=+2100.289922799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6159985a-714a-4a43-8645-9865f379b1a3-maas-api-tls") pod "maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" (UID: "6159985a-714a-4a43-8645-9865f379b1a3") : secret "maas-api-e2e-praxis-a-41bdda-serving-cert" not found Oct 30 12:29:06.680344 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.680328 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6159985a-714a-4a43-8645-9865f379b1a3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72\" (UID: \"6159985a-714a-4a43-8645-9865f379b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:29:06.682604 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.682582 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6159985a-714a-4a43-8645-9865f379b1a3-metrics-certs\") pod \"maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72\" (UID: \"6159985a-714a-4a43-8645-9865f379b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:29:06.702998 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.702977 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb7vp\" (UniqueName: \"kubernetes.io/projected/6159985a-714a-4a43-8645-9865f379b1a3-kube-api-access-qb7vp\") pod \"maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72\" (UID: \"6159985a-714a-4a43-8645-9865f379b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:29:06.729655 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.729611 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4" event={"ID":"8543e221-c552-43fc-a7f4-9d21127db4ef","Type":"ContainerStarted","Data":"7c0f88912855768cdc54687eb159a658ce86d8003bf68ceb9c24ff39170ce37a"} Oct 30 12:29:06.729847 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.729681 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4" event={"ID":"8543e221-c552-43fc-a7f4-9d21127db4ef","Type":"ContainerStarted","Data":"74f95502d7807f5fd3de861f04a0c77abced25b3f9157a14a3e6c490538f24ac"} Oct 30 12:29:06.729913 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.729865 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4" Oct 30 12:29:06.731649 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.731621 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt" event={"ID":"752943e6-f6f7-450b-9e69-0fda21ee5738","Type":"ContainerStarted","Data":"bc8565af2384f04180f1dadf4bb18a0a683c927ba078d19bb7307607ba63719c"} Oct 30 12:29:06.731780 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.731665 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt" event={"ID":"752943e6-f6f7-450b-9e69-0fda21ee5738","Type":"ContainerStarted","Data":"223d24079b2aea324d31ff279b173a00d006fb9e30d7dd69ee87baf110663d72"} Oct 30 12:29:06.732232 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.732218 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt" Oct 30 12:29:06.752001 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.751950 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:06.771431 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.771350 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4" podStartSLOduration=1.7713280980000001 podStartE2EDuration="1.771328098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:06.769461796 +0000 UTC m=+2099.879304678" watchObservedRunningTime="2026-10-30 12:29:06.771328098 +0000 UTC m=+2099.881170982" Oct 30 12:29:06.817421 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.817305 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt" podStartSLOduration=1.817283491 podStartE2EDuration="1.817283491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:06.811521774 +0000 UTC m=+2099.921364653" watchObservedRunningTime="2026-10-30 12:29:06.817283491 +0000 UTC m=+2099.927126390" Oct 30 12:29:06.964138 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:06.964106 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:07.192390 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:07.192359 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6159985a-714a-4a43-8645-9865f379b1a3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72\" (UID: \"6159985a-714a-4a43-8645-9865f379b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:29:07.195028 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:07.195000 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6159985a-714a-4a43-8645-9865f379b1a3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72\" (UID: \"6159985a-714a-4a43-8645-9865f379b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:29:07.454318 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:07.454222 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:29:07.764428 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:07.764393 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72"] Oct 30 12:29:07.765688 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:29:07.765663 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6159985a_714a_4a43_8645_9865f379b1a3.slice/crio-8e96385275172fb6bb49999423f541cf97c9bfd30095203e4900fd79acbc4b7c WatchSource:0}: Error finding container 8e96385275172fb6bb49999423f541cf97c9bfd30095203e4900fd79acbc4b7c: Status 404 returned error can't find the container with id 8e96385275172fb6bb49999423f541cf97c9bfd30095203e4900fd79acbc4b7c Oct 30 12:29:08.414535 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:08.414505 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:08.748225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:08.748190 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" event={"ID":"6159985a-714a-4a43-8645-9865f379b1a3","Type":"ContainerStarted","Data":"a9e7dc30e7451abc6275a5748b6b8a0eb7a79bca69666e262879d19f978584a6"} Oct 30 12:29:08.748441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:08.748259 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" event={"ID":"6159985a-714a-4a43-8645-9865f379b1a3","Type":"ContainerStarted","Data":"8e96385275172fb6bb49999423f541cf97c9bfd30095203e4900fd79acbc4b7c"} Oct 30 12:29:08.748441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:08.748429 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:29:08.796265 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:08.796206 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" podStartSLOduration=2.395307314 podStartE2EDuration="2.796192915s" totalImagesPullingTime="400.885601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:08.795126566 +0000 UTC m=+2101.904969450" watchObservedRunningTime="2026-10-30 12:29:08.796192915 +0000 UTC m=+2101.906035795" Oct 30 12:29:09.621804 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:09.621773 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:09.737808 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:09.737773 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:09.939910 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:09.939875 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt"] Oct 30 12:29:09.940166 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:09.940135 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" podUID="aa0c1522-725d-43f2-abd8-c37053a1f70c" containerName="istio-proxy" containerID="cri-o://f67f129ba3b96e7730f8b6c51a4b5a8c0a33b2c95a354de04f56f7b362f5832f" gracePeriod=30 Oct 30 12:29:09.983785 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:09.983751 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:10.698284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:10.698220 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:12.308942 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:12.308899 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:12.873648 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:12.873616 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:13.496741 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:13.496712 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:13.666623 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:13.666590 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:13.790233 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:13.789844 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:13.980548 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:13.980516 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:29:13.980753 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:13.980585 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" podUID="aa0c1522-725d-43f2-abd8-c37053a1f70c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:29:14.464594 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:14.464565 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:14.762859 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:14.762779 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:29:14.876494 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:14.876466 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:15.225751 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.225723 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:29:15.308627 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.308589 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-envoy\") pod \"aa0c1522-725d-43f2-abd8-c37053a1f70c\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " Oct 30 12:29:15.309005 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.308660 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-workload-certs\") pod \"aa0c1522-725d-43f2-abd8-c37053a1f70c\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " Oct 30 12:29:15.309005 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.308755 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa0c1522-725d-43f2-abd8-c37053a1f70c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa0c1522-725d-43f2-abd8-c37053a1f70c-istiod-ca-cert\") pod \"aa0c1522-725d-43f2-abd8-c37053a1f70c\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " Oct 30 12:29:15.309005 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.308854 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-credential-socket\") pod \"aa0c1522-725d-43f2-abd8-c37053a1f70c\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " Oct 30 12:29:15.309005 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.308916 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-data\") pod \"aa0c1522-725d-43f2-abd8-c37053a1f70c\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " Oct 30 12:29:15.309005 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.308955 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa0c1522-725d-43f2-abd8-c37053a1f70c-kube-api-access-k8r8n\" (UniqueName: \"kubernetes.io/projected/aa0c1522-725d-43f2-abd8-c37053a1f70c-kube-api-access-k8r8n\") pod \"aa0c1522-725d-43f2-abd8-c37053a1f70c\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " Oct 30 12:29:15.309380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.309017 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-token\" (UniqueName: \"kubernetes.io/projected/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-token\") pod \"aa0c1522-725d-43f2-abd8-c37053a1f70c\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " Oct 30 12:29:15.309380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.309098 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-podinfo\") pod \"aa0c1522-725d-43f2-abd8-c37053a1f70c\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " Oct 30 12:29:15.309380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.309170 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-workload-socket\") pod \"aa0c1522-725d-43f2-abd8-c37053a1f70c\" (UID: \"aa0c1522-725d-43f2-abd8-c37053a1f70c\") " Oct 30 12:29:15.311309 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.311210 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-workload-certs" pod "aa0c1522-725d-43f2-abd8-c37053a1f70c" (UID: "aa0c1522-725d-43f2-abd8-c37053a1f70c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:15.312845 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.312819 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-credential-socket" pod "aa0c1522-725d-43f2-abd8-c37053a1f70c" (UID: "aa0c1522-725d-43f2-abd8-c37053a1f70c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:15.312990 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.312841 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-workload-socket" pod "aa0c1522-725d-43f2-abd8-c37053a1f70c" (UID: "aa0c1522-725d-43f2-abd8-c37053a1f70c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:15.313113 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.313079 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa0c1522-725d-43f2-abd8-c37053a1f70c-istiod-ca-cert" pod "aa0c1522-725d-43f2-abd8-c37053a1f70c" (UID: "aa0c1522-725d-43f2-abd8-c37053a1f70c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:29:15.313158 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.312801 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-data" pod "aa0c1522-725d-43f2-abd8-c37053a1f70c" (UID: "aa0c1522-725d-43f2-abd8-c37053a1f70c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:15.314347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.314291 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-podinfo" pod "aa0c1522-725d-43f2-abd8-c37053a1f70c" (UID: "aa0c1522-725d-43f2-abd8-c37053a1f70c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:29:15.314469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.314431 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa0c1522-725d-43f2-abd8-c37053a1f70c-kube-api-access-k8r8n" pod "aa0c1522-725d-43f2-abd8-c37053a1f70c" (UID: "aa0c1522-725d-43f2-abd8-c37053a1f70c"). InnerVolumeSpecName "kube-api-access-k8r8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:15.317326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.315599 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-token" pod "aa0c1522-725d-43f2-abd8-c37053a1f70c" (UID: "aa0c1522-725d-43f2-abd8-c37053a1f70c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:15.317435 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.316025 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-envoy" pod "aa0c1522-725d-43f2-abd8-c37053a1f70c" (UID: "aa0c1522-725d-43f2-abd8-c37053a1f70c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:15.388637 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.388592 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:15.411397 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.411372 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:15.411397 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.411397 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:15.411588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.411411 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8r8n\" (UniqueName: \"kubernetes.io/projected/aa0c1522-725d-43f2-abd8-c37053a1f70c-kube-api-access-k8r8n\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:15.411588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.411424 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:15.411588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.411435 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:15.411588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.411447 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:15.411588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.411459 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:15.411588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.411473 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa0c1522-725d-43f2-abd8-c37053a1f70c-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:15.411588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.411484 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa0c1522-725d-43f2-abd8-c37053a1f70c-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:15.564929 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.564856 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:15.806003 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.805974 4413 generic.go:356] "Generic (PLEG): container finished" podID="aa0c1522-725d-43f2-abd8-c37053a1f70c" containerID="f67f129ba3b96e7730f8b6c51a4b5a8c0a33b2c95a354de04f56f7b362f5832f" exitCode=0 Oct 30 12:29:15.806506 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.806033 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" Oct 30 12:29:15.806506 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.806058 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" event={"ID":"aa0c1522-725d-43f2-abd8-c37053a1f70c","Type":"ContainerDied","Data":"f67f129ba3b96e7730f8b6c51a4b5a8c0a33b2c95a354de04f56f7b362f5832f"} Oct 30 12:29:15.806506 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.806112 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt" event={"ID":"aa0c1522-725d-43f2-abd8-c37053a1f70c","Type":"ContainerDied","Data":"43fd3fefc3d9ae743bda242b478a5a06926863e907e943255a474b5c8dad7fba"} Oct 30 12:29:15.806506 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.806134 4413 scope.go:122] "RemoveContainer" containerID="f67f129ba3b96e7730f8b6c51a4b5a8c0a33b2c95a354de04f56f7b362f5832f" Oct 30 12:29:15.817051 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.817036 4413 scope.go:122] "RemoveContainer" containerID="de2579e32ebcd73db1abfd137677870b35b605be3cce623e19a42358a1cdabdf" Oct 30 12:29:15.826203 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.826188 4413 scope.go:122] "RemoveContainer" containerID="f67f129ba3b96e7730f8b6c51a4b5a8c0a33b2c95a354de04f56f7b362f5832f" Oct 30 12:29:15.826483 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:29:15.826466 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67f129ba3b96e7730f8b6c51a4b5a8c0a33b2c95a354de04f56f7b362f5832f\": container with ID starting with f67f129ba3b96e7730f8b6c51a4b5a8c0a33b2c95a354de04f56f7b362f5832f not found: ID does not exist" containerID="f67f129ba3b96e7730f8b6c51a4b5a8c0a33b2c95a354de04f56f7b362f5832f" Oct 30 12:29:15.826541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.826491 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67f129ba3b96e7730f8b6c51a4b5a8c0a33b2c95a354de04f56f7b362f5832f"} err="failed to get container status \"f67f129ba3b96e7730f8b6c51a4b5a8c0a33b2c95a354de04f56f7b362f5832f\": rpc error: code = NotFound desc = could not find container \"f67f129ba3b96e7730f8b6c51a4b5a8c0a33b2c95a354de04f56f7b362f5832f\": container with ID starting with f67f129ba3b96e7730f8b6c51a4b5a8c0a33b2c95a354de04f56f7b362f5832f not found: ID does not exist" Oct 30 12:29:15.826541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.826510 4413 scope.go:122] "RemoveContainer" containerID="de2579e32ebcd73db1abfd137677870b35b605be3cce623e19a42358a1cdabdf" Oct 30 12:29:15.826738 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:29:15.826718 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2579e32ebcd73db1abfd137677870b35b605be3cce623e19a42358a1cdabdf\": container with ID starting with de2579e32ebcd73db1abfd137677870b35b605be3cce623e19a42358a1cdabdf not found: ID does not exist" containerID="de2579e32ebcd73db1abfd137677870b35b605be3cce623e19a42358a1cdabdf" Oct 30 12:29:15.826791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.826747 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2579e32ebcd73db1abfd137677870b35b605be3cce623e19a42358a1cdabdf"} err="failed to get container status \"de2579e32ebcd73db1abfd137677870b35b605be3cce623e19a42358a1cdabdf\": rpc error: code = NotFound desc = could not find container \"de2579e32ebcd73db1abfd137677870b35b605be3cce623e19a42358a1cdabdf\": container with ID starting with de2579e32ebcd73db1abfd137677870b35b605be3cce623e19a42358a1cdabdf not found: ID does not exist" Oct 30 12:29:15.858330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.858310 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt"] Oct 30 12:29:15.869653 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:15.869633 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-096834-openshift-default-7d7f756545-rqhnt"] Oct 30 12:29:16.628472 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:16.628443 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:17.008768 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:17.008684 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:17.020663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:17.020632 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa0c1522-725d-43f2-abd8-c37053a1f70c" path="/var/lib/kubelet/pods/aa0c1522-725d-43f2-abd8-c37053a1f70c/volumes" Oct 30 12:29:17.150883 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:17.150846 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:17.743595 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:17.743570 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4" Oct 30 12:29:18.709971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:18.709927 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:18.752503 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:18.752479 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt" Oct 30 12:29:18.845663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:18.845633 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:19.064350 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.064184 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh"] Oct 30 12:29:19.074891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.065065 4413 cpu_manager.go:414] "containerMap: removing container" podUID="aa0c1522-725d-43f2-abd8-c37053a1f70c" containerName="istio-proxy" Oct 30 12:29:19.074891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.065089 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa0c1522-725d-43f2-abd8-c37053a1f70c" containerName="istio-proxy" Oct 30 12:29:19.074891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.065274 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa0c1522-725d-43f2-abd8-c37053a1f70c" containerName="istio-proxy" Oct 30 12:29:19.074891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.070801 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh"] Oct 30 12:29:19.074891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.070906 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" Oct 30 12:29:19.088833 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.088799 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c44eee\"" Oct 30 12:29:19.089322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.089302 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c44eee-dockercfg-xm48l\"" Oct 30 12:29:19.133663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.133628 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:19.170688 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.170654 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjc4r\" (UniqueName: \"kubernetes.io/projected/9d96d03b-bb3c-4841-a9e5-94996131da71-kube-api-access-hjc4r\") pod \"payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh\" (UID: \"9d96d03b-bb3c-4841-a9e5-94996131da71\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" Oct 30 12:29:19.170821 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.170797 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d96d03b-bb3c-4841-a9e5-94996131da71-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh\" (UID: \"9d96d03b-bb3c-4841-a9e5-94996131da71\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" Oct 30 12:29:19.272036 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.272004 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjc4r\" (UniqueName: \"kubernetes.io/projected/9d96d03b-bb3c-4841-a9e5-94996131da71-kube-api-access-hjc4r\") pod \"payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh\" (UID: \"9d96d03b-bb3c-4841-a9e5-94996131da71\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" Oct 30 12:29:19.272195 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.272121 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d96d03b-bb3c-4841-a9e5-94996131da71-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh\" (UID: \"9d96d03b-bb3c-4841-a9e5-94996131da71\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" Oct 30 12:29:19.272894 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.272868 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d96d03b-bb3c-4841-a9e5-94996131da71-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh\" (UID: \"9d96d03b-bb3c-4841-a9e5-94996131da71\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" Oct 30 12:29:19.295064 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.295036 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:19.295178 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.295125 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjc4r\" (UniqueName: \"kubernetes.io/projected/9d96d03b-bb3c-4841-a9e5-94996131da71-kube-api-access-hjc4r\") pod \"payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh\" (UID: \"9d96d03b-bb3c-4841-a9e5-94996131da71\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" Oct 30 12:29:19.412002 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.411968 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:19.416018 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.416000 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" Oct 30 12:29:19.669556 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.669478 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:19.722675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.722648 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh"] Oct 30 12:29:19.723879 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:29:19.723852 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d96d03b_bb3c_4841_a9e5_94996131da71.slice/crio-efd0af80af247da99307999e9decd669ed822f6e362766e3ec43e053cc9c3206 WatchSource:0}: Error finding container efd0af80af247da99307999e9decd669ed822f6e362766e3ec43e053cc9c3206: Status 404 returned error can't find the container with id efd0af80af247da99307999e9decd669ed822f6e362766e3ec43e053cc9c3206 Oct 30 12:29:19.840864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.840833 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" event={"ID":"9d96d03b-bb3c-4841-a9e5-94996131da71","Type":"ContainerStarted","Data":"9a560c59841c98fe45984d5fc7533afd1c2ce741ea772eeefb5f41bedefe8d98"} Oct 30 12:29:19.841001 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.840886 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" event={"ID":"9d96d03b-bb3c-4841-a9e5-94996131da71","Type":"ContainerStarted","Data":"efd0af80af247da99307999e9decd669ed822f6e362766e3ec43e053cc9c3206"} Oct 30 12:29:19.841001 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.840922 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" Oct 30 12:29:19.884704 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:19.884655 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" podStartSLOduration=1.884642656 podStartE2EDuration="1.884642656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:19.881874331 +0000 UTC m=+2112.991717211" watchObservedRunningTime="2026-10-30 12:29:19.884642656 +0000 UTC m=+2112.994485536" Oct 30 12:29:21.240783 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.240745 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:21.348855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.348822 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn"] Oct 30 12:29:21.349558 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.349542 4413 cpu_manager.go:414] "containerMap: removing container" podUID="aa0c1522-725d-43f2-abd8-c37053a1f70c" containerName="istio-proxy" Oct 30 12:29:21.349615 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.349559 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa0c1522-725d-43f2-abd8-c37053a1f70c" containerName="istio-proxy" Oct 30 12:29:21.349676 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.349667 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa0c1522-725d-43f2-abd8-c37053a1f70c" containerName="istio-proxy" Oct 30 12:29:21.353481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.353463 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn" Oct 30 12:29:21.359401 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.359370 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:21.369328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.369202 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ec236e\"" Oct 30 12:29:21.369328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.369213 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ec236e-dockercfg-gtzwd\"" Oct 30 12:29:21.373584 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.373560 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn"] Oct 30 12:29:21.397354 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.397322 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d20b6be-6972-426c-afff-bb5ae5f93a6e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn\" (UID: \"2d20b6be-6972-426c-afff-bb5ae5f93a6e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn" Oct 30 12:29:21.397627 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.397604 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59rqs\" (UniqueName: \"kubernetes.io/projected/2d20b6be-6972-426c-afff-bb5ae5f93a6e-kube-api-access-59rqs\") pod \"payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn\" (UID: \"2d20b6be-6972-426c-afff-bb5ae5f93a6e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn" Oct 30 12:29:21.519320 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.516229 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59rqs\" (UniqueName: \"kubernetes.io/projected/2d20b6be-6972-426c-afff-bb5ae5f93a6e-kube-api-access-59rqs\") pod \"payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn\" (UID: \"2d20b6be-6972-426c-afff-bb5ae5f93a6e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn" Oct 30 12:29:21.519320 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.516362 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d20b6be-6972-426c-afff-bb5ae5f93a6e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn\" (UID: \"2d20b6be-6972-426c-afff-bb5ae5f93a6e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn" Oct 30 12:29:21.574887 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.574003 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d20b6be-6972-426c-afff-bb5ae5f93a6e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn\" (UID: \"2d20b6be-6972-426c-afff-bb5ae5f93a6e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn" Oct 30 12:29:21.584699 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.584665 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59rqs\" (UniqueName: \"kubernetes.io/projected/2d20b6be-6972-426c-afff-bb5ae5f93a6e-kube-api-access-59rqs\") pod \"payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn\" (UID: \"2d20b6be-6972-426c-afff-bb5ae5f93a6e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn" Oct 30 12:29:21.621763 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.621707 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:21.668820 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.668776 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn" Oct 30 12:29:21.982297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:21.979156 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn"] Oct 30 12:29:22.862660 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:22.862622 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn" event={"ID":"2d20b6be-6972-426c-afff-bb5ae5f93a6e","Type":"ContainerStarted","Data":"772557a91b375d2a2bdf88e7a17364d8ff66a40ad301ceeba30390d8f824572d"} Oct 30 12:29:22.863095 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:22.862805 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn" Oct 30 12:29:22.863095 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:22.862838 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn" event={"ID":"2d20b6be-6972-426c-afff-bb5ae5f93a6e","Type":"ContainerStarted","Data":"b480c230228c5110ef276c71e61b8feefbe05ec24f2ea3601af8c7943aa5121c"} Oct 30 12:29:22.905007 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:22.904957 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn" podStartSLOduration=1.904943864 podStartE2EDuration="1.904943864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:22.902132349 +0000 UTC m=+2116.011975230" watchObservedRunningTime="2026-10-30 12:29:22.904943864 +0000 UTC m=+2116.014786746" Oct 30 12:29:23.522703 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:23.522667 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:25.143661 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.143627 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5"] Oct 30 12:29:25.148156 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.148141 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" Oct 30 12:29:25.165134 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.165105 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-7f78d4-serving-cert\"" Oct 30 12:29:25.183289 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.181905 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5"] Oct 30 12:29:25.188192 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.188166 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-metrics-certs\") pod \"maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5\" (UID: \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" Oct 30 12:29:25.188337 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.188321 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5\" (UID: \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" Oct 30 12:29:25.188451 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.188437 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkx8s\" (UniqueName: \"kubernetes.io/projected/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-kube-api-access-gkx8s\") pod \"maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5\" (UID: \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" Oct 30 12:29:25.188498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.188463 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5\" (UID: \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" Oct 30 12:29:25.231927 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.231896 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:25.289951 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.289926 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkx8s\" (UniqueName: \"kubernetes.io/projected/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-kube-api-access-gkx8s\") pod \"maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5\" (UID: \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" Oct 30 12:29:25.290085 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.289960 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5\" (UID: \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" Oct 30 12:29:25.290085 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.290026 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-metrics-certs\") pod \"maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5\" (UID: \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" Oct 30 12:29:25.290161 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.290091 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5\" (UID: \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" Oct 30 12:29:25.290735 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.290709 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5\" (UID: \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" Oct 30 12:29:25.292536 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.292511 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-metrics-certs\") pod \"maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5\" (UID: \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" Oct 30 12:29:25.292622 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.292578 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5\" (UID: \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" Oct 30 12:29:25.312686 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.312666 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkx8s\" (UniqueName: \"kubernetes.io/projected/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-kube-api-access-gkx8s\") pod \"maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5\" (UID: \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" Oct 30 12:29:25.350541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.350507 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:25.462477 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.462289 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" Oct 30 12:29:25.465064 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.465019 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:25.784096 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.784070 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5"] Oct 30 12:29:25.785719 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:29:25.785687 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fa9579_e6a6_4c1d_a867_da1bd42fa91b.slice/crio-e795a312553fef2623e2448e974cf692402e5423b4639fb11559d7cc7ed50a15 WatchSource:0}: Error finding container e795a312553fef2623e2448e974cf692402e5423b4639fb11559d7cc7ed50a15: Status 404 returned error can't find the container with id e795a312553fef2623e2448e974cf692402e5423b4639fb11559d7cc7ed50a15 Oct 30 12:29:25.882044 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:25.882014 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" event={"ID":"51fa9579-e6a6-4c1d-a867-da1bd42fa91b","Type":"ContainerStarted","Data":"e795a312553fef2623e2448e974cf692402e5423b4639fb11559d7cc7ed50a15"} Oct 30 12:29:26.889983 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:26.889947 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" event={"ID":"51fa9579-e6a6-4c1d-a867-da1bd42fa91b","Type":"ContainerStarted","Data":"f336216edbc704d40526487ecf7300f539b95d72ff2d32582a0b16ee6100746f"} Oct 30 12:29:26.890474 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:26.890173 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" Oct 30 12:29:26.934707 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:26.934660 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" podStartSLOduration=1.56515394 podStartE2EDuration="1.934647259s" totalImagesPullingTime="369.493319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:26.931484865 +0000 UTC m=+2120.041327746" watchObservedRunningTime="2026-10-30 12:29:26.934647259 +0000 UTC m=+2120.044490139" Oct 30 12:29:27.175565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:27.175489 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:29.351207 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:29.351170 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:30.850973 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:30.850944 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" Oct 30 12:29:30.947073 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:30.947043 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:32.755063 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:32.755004 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:32.903073 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:32.903049 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" Oct 30 12:29:33.871527 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:33.871498 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn" Oct 30 12:29:35.168436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:35.168401 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:35.814815 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:29:35.814774 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:29:35.959181 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:35.959148 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm"] Oct 30 12:29:35.959436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:35.959409 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" podUID="18e35374-76c0-476a-bfcf-0a3e253718ee" containerName="istio-proxy" containerID="cri-o://05197d46f7b85982dd412718fc53b721aebc2f87439b4ace83eea8d6fab9a01b" gracePeriod=30 Oct 30 12:29:36.934565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:36.934532 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:37.056396 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:37.056365 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:37.400816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:37.400787 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:38.673102 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:38.673059 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:38.794920 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:38.794887 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:39.274888 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:39.274837 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:39.396588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:39.396548 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:40.511194 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:40.511144 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:40.626434 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:40.626389 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:41.232646 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.232576 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:29:41.315691 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.315663 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e35374-76c0-476a-bfcf-0a3e253718ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e35374-76c0-476a-bfcf-0a3e253718ee-istiod-ca-cert\") pod \"18e35374-76c0-476a-bfcf-0a3e253718ee\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " Oct 30 12:29:41.315873 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.315705 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-envoy\") pod \"18e35374-76c0-476a-bfcf-0a3e253718ee\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " Oct 30 12:29:41.315873 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.315813 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-credential-socket\") pod \"18e35374-76c0-476a-bfcf-0a3e253718ee\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " Oct 30 12:29:41.315873 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.315874 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e35374-76c0-476a-bfcf-0a3e253718ee-kube-api-access-rf2dj\" (UniqueName: \"kubernetes.io/projected/18e35374-76c0-476a-bfcf-0a3e253718ee-kube-api-access-rf2dj\") pod \"18e35374-76c0-476a-bfcf-0a3e253718ee\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " Oct 30 12:29:41.316066 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.315892 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-workload-socket\") pod \"18e35374-76c0-476a-bfcf-0a3e253718ee\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " Oct 30 12:29:41.316066 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.315953 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-data\") pod \"18e35374-76c0-476a-bfcf-0a3e253718ee\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " Oct 30 12:29:41.316066 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.315989 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-podinfo\") pod \"18e35374-76c0-476a-bfcf-0a3e253718ee\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " Oct 30 12:29:41.316210 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.316068 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e35374-76c0-476a-bfcf-0a3e253718ee-istiod-ca-cert" pod "18e35374-76c0-476a-bfcf-0a3e253718ee" (UID: "18e35374-76c0-476a-bfcf-0a3e253718ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:29:41.316210 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.316088 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-token\" (UniqueName: \"kubernetes.io/projected/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-token\") pod \"18e35374-76c0-476a-bfcf-0a3e253718ee\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " Oct 30 12:29:41.316210 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.316124 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-workload-certs\") pod \"18e35374-76c0-476a-bfcf-0a3e253718ee\" (UID: \"18e35374-76c0-476a-bfcf-0a3e253718ee\") " Oct 30 12:29:41.317067 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.316798 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-data" pod "18e35374-76c0-476a-bfcf-0a3e253718ee" (UID: "18e35374-76c0-476a-bfcf-0a3e253718ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:41.317067 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.317014 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-workload-certs" pod "18e35374-76c0-476a-bfcf-0a3e253718ee" (UID: "18e35374-76c0-476a-bfcf-0a3e253718ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:41.317302 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.317125 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-credential-socket" pod "18e35374-76c0-476a-bfcf-0a3e253718ee" (UID: "18e35374-76c0-476a-bfcf-0a3e253718ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:41.317302 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.317191 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-workload-socket" pod "18e35374-76c0-476a-bfcf-0a3e253718ee" (UID: "18e35374-76c0-476a-bfcf-0a3e253718ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:41.317746 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.317636 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:41.317746 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.317666 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:41.317746 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.317682 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:41.317746 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.317710 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:41.317746 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.317724 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e35374-76c0-476a-bfcf-0a3e253718ee-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:41.318609 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.318573 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e35374-76c0-476a-bfcf-0a3e253718ee-kube-api-access-rf2dj" pod "18e35374-76c0-476a-bfcf-0a3e253718ee" (UID: "18e35374-76c0-476a-bfcf-0a3e253718ee"). InnerVolumeSpecName "kube-api-access-rf2dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:41.318926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.318890 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-podinfo" pod "18e35374-76c0-476a-bfcf-0a3e253718ee" (UID: "18e35374-76c0-476a-bfcf-0a3e253718ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:29:41.319004 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.318944 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-token" pod "18e35374-76c0-476a-bfcf-0a3e253718ee" (UID: "18e35374-76c0-476a-bfcf-0a3e253718ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:29:41.319004 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.318956 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-envoy" pod "18e35374-76c0-476a-bfcf-0a3e253718ee" (UID: "18e35374-76c0-476a-bfcf-0a3e253718ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:29:41.419078 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.419041 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:41.419078 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.419076 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf2dj\" (UniqueName: \"kubernetes.io/projected/18e35374-76c0-476a-bfcf-0a3e253718ee-kube-api-access-rf2dj\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:41.419078 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.419087 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:41.419078 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.419097 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e35374-76c0-476a-bfcf-0a3e253718ee-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:29:41.995713 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.995675 4413 generic.go:356] "Generic (PLEG): container finished" podID="18e35374-76c0-476a-bfcf-0a3e253718ee" containerID="05197d46f7b85982dd412718fc53b721aebc2f87439b4ace83eea8d6fab9a01b" exitCode=0 Oct 30 12:29:41.996188 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.995840 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" event={"ID":"18e35374-76c0-476a-bfcf-0a3e253718ee","Type":"ContainerDied","Data":"05197d46f7b85982dd412718fc53b721aebc2f87439b4ace83eea8d6fab9a01b"} Oct 30 12:29:41.996188 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.995850 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" Oct 30 12:29:41.996188 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.995896 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm" event={"ID":"18e35374-76c0-476a-bfcf-0a3e253718ee","Type":"ContainerDied","Data":"dd0ae584e7d61b669a3dceedc8b696d05ecd3ef92dc68628f5a0d9deb69b4988"} Oct 30 12:29:41.996188 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:41.995926 4413 scope.go:122] "RemoveContainer" containerID="05197d46f7b85982dd412718fc53b721aebc2f87439b4ace83eea8d6fab9a01b" Oct 30 12:29:42.011653 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:42.011630 4413 scope.go:122] "RemoveContainer" containerID="05197d46f7b85982dd412718fc53b721aebc2f87439b4ace83eea8d6fab9a01b" Oct 30 12:29:42.013121 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:29:42.013093 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05197d46f7b85982dd412718fc53b721aebc2f87439b4ace83eea8d6fab9a01b\": container with ID starting with 05197d46f7b85982dd412718fc53b721aebc2f87439b4ace83eea8d6fab9a01b not found: ID does not exist" containerID="05197d46f7b85982dd412718fc53b721aebc2f87439b4ace83eea8d6fab9a01b" Oct 30 12:29:42.013347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:42.013131 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05197d46f7b85982dd412718fc53b721aebc2f87439b4ace83eea8d6fab9a01b"} err="failed to get container status \"05197d46f7b85982dd412718fc53b721aebc2f87439b4ace83eea8d6fab9a01b\": rpc error: code = NotFound desc = could not find container \"05197d46f7b85982dd412718fc53b721aebc2f87439b4ace83eea8d6fab9a01b\": container with ID starting with 05197d46f7b85982dd412718fc53b721aebc2f87439b4ace83eea8d6fab9a01b not found: ID does not exist" Oct 30 12:29:42.075935 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:42.075905 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm"] Oct 30 12:29:42.087059 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:42.087033 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-49xmm"] Oct 30 12:29:42.465458 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:42.465405 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:43.020982 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:43.020951 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e35374-76c0-476a-bfcf-0a3e253718ee" path="/var/lib/kubelet/pods/18e35374-76c0-476a-bfcf-0a3e253718ee/volumes" Oct 30 12:29:43.531779 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:43.531738 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:43.648238 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:43.648202 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:44.873688 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:44.873651 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:44.986078 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:44.986037 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:45.101132 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:45.101088 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:45.952011 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:45.951980 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:46.074689 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:46.074656 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:46.181265 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:46.181223 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4"] Oct 30 12:29:46.181981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:46.181959 4413 cpu_manager.go:414] "containerMap: removing container" podUID="18e35374-76c0-476a-bfcf-0a3e253718ee" containerName="istio-proxy" Oct 30 12:29:46.181981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:46.181978 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e35374-76c0-476a-bfcf-0a3e253718ee" containerName="istio-proxy" Oct 30 12:29:46.182108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:46.182097 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e35374-76c0-476a-bfcf-0a3e253718ee" containerName="istio-proxy" Oct 30 12:29:46.187454 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:46.187427 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" Oct 30 12:29:46.243333 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:46.243302 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-1e9675-dockercfg-2qvxs\"" Oct 30 12:29:46.284469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:46.284438 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-1e9675\"" Oct 30 12:29:46.306690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:46.306654 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4"] Oct 30 12:29:46.399748 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:46.399715 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-654rt\" (UniqueName: \"kubernetes.io/projected/78cbff45-e07e-4847-af31-667665133aeb-kube-api-access-654rt\") pod \"payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4\" (UID: \"78cbff45-e07e-4847-af31-667665133aeb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" Oct 30 12:29:46.400058 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:46.399991 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78cbff45-e07e-4847-af31-667665133aeb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4\" (UID: \"78cbff45-e07e-4847-af31-667665133aeb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" Oct 30 12:29:46.501507 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:46.501421 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78cbff45-e07e-4847-af31-667665133aeb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4\" (UID: \"78cbff45-e07e-4847-af31-667665133aeb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" Oct 30 12:29:46.501682 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:46.501546 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-654rt\" (UniqueName: \"kubernetes.io/projected/78cbff45-e07e-4847-af31-667665133aeb-kube-api-access-654rt\") pod \"payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4\" (UID: \"78cbff45-e07e-4847-af31-667665133aeb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" Oct 30 12:29:46.502175 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:46.502151 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78cbff45-e07e-4847-af31-667665133aeb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4\" (UID: \"78cbff45-e07e-4847-af31-667665133aeb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" Oct 30 12:29:46.524922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:46.524890 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-654rt\" (UniqueName: \"kubernetes.io/projected/78cbff45-e07e-4847-af31-667665133aeb-kube-api-access-654rt\") pod \"payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4\" (UID: \"78cbff45-e07e-4847-af31-667665133aeb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" Oct 30 12:29:46.544021 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:46.544000 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" Oct 30 12:29:46.869335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:46.864151 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4"] Oct 30 12:29:47.035716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:47.035680 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" event={"ID":"78cbff45-e07e-4847-af31-667665133aeb","Type":"ContainerStarted","Data":"3156188d4887b5123f85ddc62e050f10e61bee989e032c80a01ead4ea540d854"} Oct 30 12:29:47.036200 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:47.035749 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" Oct 30 12:29:47.036200 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:47.035770 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" event={"ID":"78cbff45-e07e-4847-af31-667665133aeb","Type":"ContainerStarted","Data":"0416d69315863e5ab8e7800094e963bdc192d429021eaf3fb8c1cb07f215224e"} Oct 30 12:29:47.080436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:47.080381 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" podStartSLOduration=1.080367307 podStartE2EDuration="1.080367307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:29:47.07707493 +0000 UTC m=+2140.186917814" watchObservedRunningTime="2026-10-30 12:29:47.080367307 +0000 UTC m=+2140.190210188" Oct 30 12:29:47.593244 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:47.593212 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:49.461154 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:49.461114 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:51.125349 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:51.125315 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:52.644625 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:52.644588 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:54.441305 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:54.441238 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:56.688216 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:56.688181 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm"] Oct 30 12:29:56.688785 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:56.688412 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" podUID="57785552-cd5b-48b5-b4fc-a62d21e76b8d" containerName="istio-proxy" containerID="cri-o://d32f2cf6cb59a8ea5cafa3fc53bbb34239885f19543c9a70825d859916ecccb5" gracePeriod=30 Oct 30 12:29:56.775552 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:56.775517 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:57.978758 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:57.978727 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:58.045115 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:58.045084 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" Oct 30 12:29:58.107722 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:58.107694 4413 generic.go:356] "Generic (PLEG): container finished" podID="0250c9a9-b527-47e2-8d57-f221fcbd8e19" containerID="6f2735db2e680fe4fc391bee550fba4ad193823dbad37128da8fe2bbefc96cb2" exitCode=0 Oct 30 12:29:58.107873 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:58.107770 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" event={"ID":"0250c9a9-b527-47e2-8d57-f221fcbd8e19","Type":"ContainerDied","Data":"6f2735db2e680fe4fc391bee550fba4ad193823dbad37128da8fe2bbefc96cb2"} Oct 30 12:29:58.108296 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:58.108242 4413 scope.go:122] "RemoveContainer" containerID="6f2735db2e680fe4fc391bee550fba4ad193823dbad37128da8fe2bbefc96cb2" Oct 30 12:29:58.109904 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:58.109870 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:29:58.110024 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:58.109952 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:29:58.110024 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:58.109991 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:29:58.455963 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:58.455925 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:59.124838 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:59.124796 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" event={"ID":"0250c9a9-b527-47e2-8d57-f221fcbd8e19","Type":"ContainerStarted","Data":"7d4e0571053faf2806ac90492208c9dbc0e2f8ae4e384ac966355c52be042886"} Oct 30 12:29:59.418487 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:59.418393 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4"] Oct 30 12:29:59.450903 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:59.450871 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt"] Oct 30 12:29:59.565580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:59.565538 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:59.741927 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:59.741820 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:29:59.859344 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:29:59.859310 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:00.023529 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:00.023439 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:30:00.024820 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:00.024798 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 12:30:00.024940 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:00.024860 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" podUID="0250c9a9-b527-47e2-8d57-f221fcbd8e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 12:30:00.072461 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:00.072432 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:00.256122 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:00.253213 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz"] Oct 30 12:30:00.262303 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:00.258517 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" Oct 30 12:30:00.280025 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:00.279949 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz"] Oct 30 12:30:00.312275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:00.292520 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:30:00.371792 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:00.365366 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d164b031-7ba1-48d2-9890-01e3e941ad3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz\" (UID: \"d164b031-7ba1-48d2-9890-01e3e941ad3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" Oct 30 12:30:00.372243 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:00.372210 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpwqj\" (UniqueName: \"kubernetes.io/projected/d164b031-7ba1-48d2-9890-01e3e941ad3e-kube-api-access-xpwqj\") pod \"maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz\" (UID: \"d164b031-7ba1-48d2-9890-01e3e941ad3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" Oct 30 12:30:00.473427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:00.473396 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d164b031-7ba1-48d2-9890-01e3e941ad3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz\" (UID: \"d164b031-7ba1-48d2-9890-01e3e941ad3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" Oct 30 12:30:00.473610 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:00.473467 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpwqj\" (UniqueName: \"kubernetes.io/projected/d164b031-7ba1-48d2-9890-01e3e941ad3e-kube-api-access-xpwqj\") pod \"maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz\" (UID: \"d164b031-7ba1-48d2-9890-01e3e941ad3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" Oct 30 12:30:00.474032 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:00.474010 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d164b031-7ba1-48d2-9890-01e3e941ad3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz\" (UID: \"d164b031-7ba1-48d2-9890-01e3e941ad3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" Oct 30 12:30:00.517725 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:00.517686 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpwqj\" (UniqueName: \"kubernetes.io/projected/d164b031-7ba1-48d2-9890-01e3e941ad3e-kube-api-access-xpwqj\") pod \"maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz\" (UID: \"d164b031-7ba1-48d2-9890-01e3e941ad3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" Oct 30 12:30:00.689529 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:00.689489 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" Oct 30 12:30:01.027005 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:01.026497 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 12:30:01.027005 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:01.026557 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" podUID="0250c9a9-b527-47e2-8d57-f221fcbd8e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 12:30:01.044429 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:30:01.044377 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd164b031_7ba1_48d2_9890_01e3e941ad3e.slice/crio-57704dafa789170a913030fb14f4a8b5ed47766fde2c81eb500ba119b0bc2356 WatchSource:0}: Error finding container 57704dafa789170a913030fb14f4a8b5ed47766fde2c81eb500ba119b0bc2356: Status 404 returned error can't find the container with id 57704dafa789170a913030fb14f4a8b5ed47766fde2c81eb500ba119b0bc2356 Oct 30 12:30:01.044659 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:01.044636 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz"] Oct 30 12:30:01.141472 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:01.141443 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" event={"ID":"d164b031-7ba1-48d2-9890-01e3e941ad3e","Type":"ContainerStarted","Data":"57704dafa789170a913030fb14f4a8b5ed47766fde2c81eb500ba119b0bc2356"} Oct 30 12:30:01.628607 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:01.628559 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:30:01.629156 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:01.628625 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" podUID="57785552-cd5b-48b5-b4fc-a62d21e76b8d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:30:01.689932 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:01.689897 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:30:02.022915 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.022736 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 12:30:02.022915 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.022793 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" podUID="0250c9a9-b527-47e2-8d57-f221fcbd8e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 12:30:02.030654 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.030619 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:30:02.124377 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.124327 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-workload-certs\") pod \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " Oct 30 12:30:02.124672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.124643 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-credential-socket\") pod \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " Oct 30 12:30:02.124672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.124691 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-workload-certs" pod "57785552-cd5b-48b5-b4fc-a62d21e76b8d" (UID: "57785552-cd5b-48b5-b4fc-a62d21e76b8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:02.125048 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.124944 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-credential-socket" pod "57785552-cd5b-48b5-b4fc-a62d21e76b8d" (UID: "57785552-cd5b-48b5-b4fc-a62d21e76b8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:02.125048 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.125027 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-envoy\") pod \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " Oct 30 12:30:02.125212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.125149 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-data\") pod \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " Oct 30 12:30:02.125440 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.125409 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-data" pod "57785552-cd5b-48b5-b4fc-a62d21e76b8d" (UID: "57785552-cd5b-48b5-b4fc-a62d21e76b8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:02.125583 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.125180 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istiod-ca-cert\") pod \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " Oct 30 12:30:02.125852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.125825 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istiod-ca-cert" pod "57785552-cd5b-48b5-b4fc-a62d21e76b8d" (UID: "57785552-cd5b-48b5-b4fc-a62d21e76b8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:02.126027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.125829 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57785552-cd5b-48b5-b4fc-a62d21e76b8d-kube-api-access-qfph9\" (UniqueName: \"kubernetes.io/projected/57785552-cd5b-48b5-b4fc-a62d21e76b8d-kube-api-access-qfph9\") pod \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " Oct 30 12:30:02.126027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.125945 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-podinfo\") pod \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " Oct 30 12:30:02.126027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.126004 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-workload-socket\") pod \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " Oct 30 12:30:02.126313 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.126132 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-token\" (UniqueName: \"kubernetes.io/projected/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-token\") pod \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\" (UID: \"57785552-cd5b-48b5-b4fc-a62d21e76b8d\") " Oct 30 12:30:02.127141 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.127113 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:02.127653 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.127417 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:02.127653 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.127574 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:02.127653 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.127593 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:02.127653 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.127602 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-workload-socket" pod "57785552-cd5b-48b5-b4fc-a62d21e76b8d" (UID: "57785552-cd5b-48b5-b4fc-a62d21e76b8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:02.136177 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.134711 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-envoy" pod "57785552-cd5b-48b5-b4fc-a62d21e76b8d" (UID: "57785552-cd5b-48b5-b4fc-a62d21e76b8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:02.136177 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.134738 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57785552-cd5b-48b5-b4fc-a62d21e76b8d-kube-api-access-qfph9" pod "57785552-cd5b-48b5-b4fc-a62d21e76b8d" (UID: "57785552-cd5b-48b5-b4fc-a62d21e76b8d"). InnerVolumeSpecName "kube-api-access-qfph9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:02.137238 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.137198 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-token" pod "57785552-cd5b-48b5-b4fc-a62d21e76b8d" (UID: "57785552-cd5b-48b5-b4fc-a62d21e76b8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:02.138309 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.138286 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-podinfo" pod "57785552-cd5b-48b5-b4fc-a62d21e76b8d" (UID: "57785552-cd5b-48b5-b4fc-a62d21e76b8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:30:02.153118 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.153082 4413 generic.go:356] "Generic (PLEG): container finished" podID="57785552-cd5b-48b5-b4fc-a62d21e76b8d" containerID="d32f2cf6cb59a8ea5cafa3fc53bbb34239885f19543c9a70825d859916ecccb5" exitCode=0 Oct 30 12:30:02.153322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.153155 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" Oct 30 12:30:02.153322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.153170 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" event={"ID":"57785552-cd5b-48b5-b4fc-a62d21e76b8d","Type":"ContainerDied","Data":"d32f2cf6cb59a8ea5cafa3fc53bbb34239885f19543c9a70825d859916ecccb5"} Oct 30 12:30:02.153322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.153234 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm" event={"ID":"57785552-cd5b-48b5-b4fc-a62d21e76b8d","Type":"ContainerDied","Data":"8694d9ab5150b314f69660e6db2d3f43ebb26642687e8753f7a131d2287d3a86"} Oct 30 12:30:02.153322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.153292 4413 scope.go:122] "RemoveContainer" containerID="d32f2cf6cb59a8ea5cafa3fc53bbb34239885f19543c9a70825d859916ecccb5" Oct 30 12:30:02.167388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.167359 4413 scope.go:122] "RemoveContainer" containerID="d32f2cf6cb59a8ea5cafa3fc53bbb34239885f19543c9a70825d859916ecccb5" Oct 30 12:30:02.167881 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:30:02.167848 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d32f2cf6cb59a8ea5cafa3fc53bbb34239885f19543c9a70825d859916ecccb5\": container with ID starting with d32f2cf6cb59a8ea5cafa3fc53bbb34239885f19543c9a70825d859916ecccb5 not found: ID does not exist" containerID="d32f2cf6cb59a8ea5cafa3fc53bbb34239885f19543c9a70825d859916ecccb5" Oct 30 12:30:02.168019 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.167894 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d32f2cf6cb59a8ea5cafa3fc53bbb34239885f19543c9a70825d859916ecccb5"} err="failed to get container status \"d32f2cf6cb59a8ea5cafa3fc53bbb34239885f19543c9a70825d859916ecccb5\": rpc error: code = NotFound desc = could not find container \"d32f2cf6cb59a8ea5cafa3fc53bbb34239885f19543c9a70825d859916ecccb5\": container with ID starting with d32f2cf6cb59a8ea5cafa3fc53bbb34239885f19543c9a70825d859916ecccb5 not found: ID does not exist" Oct 30 12:30:02.237479 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.237444 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfph9\" (UniqueName: \"kubernetes.io/projected/57785552-cd5b-48b5-b4fc-a62d21e76b8d-kube-api-access-qfph9\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:02.237479 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.237481 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:02.237717 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.237498 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:02.237717 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.237512 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:02.237717 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.237525 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57785552-cd5b-48b5-b4fc-a62d21e76b8d-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:02.238956 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.238936 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm"] Oct 30 12:30:02.243233 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.243206 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-bfcvm"] Oct 30 12:30:02.453453 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:02.453360 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:03.023133 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:03.023102 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 12:30:03.023538 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:03.023169 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" podUID="0250c9a9-b527-47e2-8d57-f221fcbd8e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 12:30:03.028333 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:03.028305 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57785552-cd5b-48b5-b4fc-a62d21e76b8d" path="/var/lib/kubelet/pods/57785552-cd5b-48b5-b4fc-a62d21e76b8d/volumes" Oct 30 12:30:04.023674 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.023634 4413 patch_prober.go:29] interesting pod/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 12:30:04.024064 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.023701 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" podUID="0250c9a9-b527-47e2-8d57-f221fcbd8e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 12:30:04.076778 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.076743 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:04.419342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.419284 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4" podUID="8543e221-c552-43fc-a7f4-9d21127db4ef" containerName="payload-processing" containerID="cri-o://7c0f88912855768cdc54687eb159a658ce86d8003bf68ceb9c24ff39170ce37a" gracePeriod=25 Oct 30 12:30:04.452260 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.452202 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt" podUID="752943e6-f6f7-450b-9e69-0fda21ee5738" containerName="payload-pre-processing" containerID="cri-o://bc8565af2384f04180f1dadf4bb18a0a683c927ba078d19bb7307607ba63719c" gracePeriod=25 Oct 30 12:30:04.775850 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.775829 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4" Oct 30 12:30:04.784382 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.784351 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt" Oct 30 12:30:04.790733 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.790703 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8543e221-c552-43fc-a7f4-9d21127db4ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8543e221-c552-43fc-a7f4-9d21127db4ef-plugins-config-volume\") pod \"8543e221-c552-43fc-a7f4-9d21127db4ef\" (UID: \"8543e221-c552-43fc-a7f4-9d21127db4ef\") " Oct 30 12:30:04.790733 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.790736 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752943e6-f6f7-450b-9e69-0fda21ee5738-kube-api-access-7zms6\" (UniqueName: \"kubernetes.io/projected/752943e6-f6f7-450b-9e69-0fda21ee5738-kube-api-access-7zms6\") pod \"752943e6-f6f7-450b-9e69-0fda21ee5738\" (UID: \"752943e6-f6f7-450b-9e69-0fda21ee5738\") " Oct 30 12:30:04.790953 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.790802 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8543e221-c552-43fc-a7f4-9d21127db4ef-kube-api-access-p4c8p\" (UniqueName: \"kubernetes.io/projected/8543e221-c552-43fc-a7f4-9d21127db4ef-kube-api-access-p4c8p\") pod \"8543e221-c552-43fc-a7f4-9d21127db4ef\" (UID: \"8543e221-c552-43fc-a7f4-9d21127db4ef\") " Oct 30 12:30:04.790953 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.790944 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752943e6-f6f7-450b-9e69-0fda21ee5738-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752943e6-f6f7-450b-9e69-0fda21ee5738-plugins-config-volume\") pod \"752943e6-f6f7-450b-9e69-0fda21ee5738\" (UID: \"752943e6-f6f7-450b-9e69-0fda21ee5738\") " Oct 30 12:30:04.791153 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.791126 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8543e221-c552-43fc-a7f4-9d21127db4ef-plugins-config-volume" pod "8543e221-c552-43fc-a7f4-9d21127db4ef" (UID: "8543e221-c552-43fc-a7f4-9d21127db4ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:04.791646 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.791620 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752943e6-f6f7-450b-9e69-0fda21ee5738-plugins-config-volume" pod "752943e6-f6f7-450b-9e69-0fda21ee5738" (UID: "752943e6-f6f7-450b-9e69-0fda21ee5738"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:04.791787 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.791673 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8543e221-c552-43fc-a7f4-9d21127db4ef-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:04.793519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.793492 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752943e6-f6f7-450b-9e69-0fda21ee5738-kube-api-access-7zms6" pod "752943e6-f6f7-450b-9e69-0fda21ee5738" (UID: "752943e6-f6f7-450b-9e69-0fda21ee5738"). InnerVolumeSpecName "kube-api-access-7zms6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:04.793636 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.793555 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8543e221-c552-43fc-a7f4-9d21127db4ef-kube-api-access-p4c8p" pod "8543e221-c552-43fc-a7f4-9d21127db4ef" (UID: "8543e221-c552-43fc-a7f4-9d21127db4ef"). InnerVolumeSpecName "kube-api-access-p4c8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:04.893020 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.892978 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zms6\" (UniqueName: \"kubernetes.io/projected/752943e6-f6f7-450b-9e69-0fda21ee5738-kube-api-access-7zms6\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:04.893185 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.893027 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4c8p\" (UniqueName: \"kubernetes.io/projected/8543e221-c552-43fc-a7f4-9d21127db4ef-kube-api-access-p4c8p\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:04.893185 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:04.893044 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/752943e6-f6f7-450b-9e69-0fda21ee5738-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:05.027614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.027576 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:30:05.029002 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.028979 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:30:05.181166 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.181134 4413 generic.go:356] "Generic (PLEG): container finished" podID="8543e221-c552-43fc-a7f4-9d21127db4ef" containerID="7c0f88912855768cdc54687eb159a658ce86d8003bf68ceb9c24ff39170ce37a" exitCode=0 Oct 30 12:30:05.181380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.181243 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4" event={"ID":"8543e221-c552-43fc-a7f4-9d21127db4ef","Type":"ContainerDied","Data":"7c0f88912855768cdc54687eb159a658ce86d8003bf68ceb9c24ff39170ce37a"} Oct 30 12:30:05.181380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.181244 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4" Oct 30 12:30:05.181380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.181326 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4" event={"ID":"8543e221-c552-43fc-a7f4-9d21127db4ef","Type":"ContainerDied","Data":"74f95502d7807f5fd3de861f04a0c77abced25b3f9157a14a3e6c490538f24ac"} Oct 30 12:30:05.181380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.181356 4413 scope.go:122] "RemoveContainer" containerID="7c0f88912855768cdc54687eb159a658ce86d8003bf68ceb9c24ff39170ce37a" Oct 30 12:30:05.183053 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.183022 4413 generic.go:356] "Generic (PLEG): container finished" podID="752943e6-f6f7-450b-9e69-0fda21ee5738" containerID="bc8565af2384f04180f1dadf4bb18a0a683c927ba078d19bb7307607ba63719c" exitCode=0 Oct 30 12:30:05.183189 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.183090 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt" Oct 30 12:30:05.183189 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.183107 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt" event={"ID":"752943e6-f6f7-450b-9e69-0fda21ee5738","Type":"ContainerDied","Data":"bc8565af2384f04180f1dadf4bb18a0a683c927ba078d19bb7307607ba63719c"} Oct 30 12:30:05.183358 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.183190 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt" event={"ID":"752943e6-f6f7-450b-9e69-0fda21ee5738","Type":"ContainerDied","Data":"223d24079b2aea324d31ff279b173a00d006fb9e30d7dd69ee87baf110663d72"} Oct 30 12:30:05.185792 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.185759 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" event={"ID":"d164b031-7ba1-48d2-9890-01e3e941ad3e","Type":"ContainerStarted","Data":"bed5d763a727e55ae983318dd8daec3ed2a6119b6695b92aaf3351b32ca19993"} Oct 30 12:30:05.194346 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.194317 4413 scope.go:122] "RemoveContainer" containerID="7c0f88912855768cdc54687eb159a658ce86d8003bf68ceb9c24ff39170ce37a" Oct 30 12:30:05.194773 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:30:05.194747 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c0f88912855768cdc54687eb159a658ce86d8003bf68ceb9c24ff39170ce37a\": container with ID starting with 7c0f88912855768cdc54687eb159a658ce86d8003bf68ceb9c24ff39170ce37a not found: ID does not exist" containerID="7c0f88912855768cdc54687eb159a658ce86d8003bf68ceb9c24ff39170ce37a" Oct 30 12:30:05.194899 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.194783 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c0f88912855768cdc54687eb159a658ce86d8003bf68ceb9c24ff39170ce37a"} err="failed to get container status \"7c0f88912855768cdc54687eb159a658ce86d8003bf68ceb9c24ff39170ce37a\": rpc error: code = NotFound desc = could not find container \"7c0f88912855768cdc54687eb159a658ce86d8003bf68ceb9c24ff39170ce37a\": container with ID starting with 7c0f88912855768cdc54687eb159a658ce86d8003bf68ceb9c24ff39170ce37a not found: ID does not exist" Oct 30 12:30:05.194899 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.194808 4413 scope.go:122] "RemoveContainer" containerID="bc8565af2384f04180f1dadf4bb18a0a683c927ba078d19bb7307607ba63719c" Oct 30 12:30:05.206855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.206828 4413 scope.go:122] "RemoveContainer" containerID="bc8565af2384f04180f1dadf4bb18a0a683c927ba078d19bb7307607ba63719c" Oct 30 12:30:05.207241 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:30:05.207216 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc8565af2384f04180f1dadf4bb18a0a683c927ba078d19bb7307607ba63719c\": container with ID starting with bc8565af2384f04180f1dadf4bb18a0a683c927ba078d19bb7307607ba63719c not found: ID does not exist" containerID="bc8565af2384f04180f1dadf4bb18a0a683c927ba078d19bb7307607ba63719c" Oct 30 12:30:05.207323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.207271 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc8565af2384f04180f1dadf4bb18a0a683c927ba078d19bb7307607ba63719c"} err="failed to get container status \"bc8565af2384f04180f1dadf4bb18a0a683c927ba078d19bb7307607ba63719c\": rpc error: code = NotFound desc = could not find container \"bc8565af2384f04180f1dadf4bb18a0a683c927ba078d19bb7307607ba63719c\": container with ID starting with bc8565af2384f04180f1dadf4bb18a0a683c927ba078d19bb7307607ba63719c not found: ID does not exist" Oct 30 12:30:05.238000 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.237955 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt"] Oct 30 12:30:05.241385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.241352 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7f78d4-7446cb464c-lsdwt"] Oct 30 12:30:05.282195 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.282076 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" podStartSLOduration=1.828377978 podStartE2EDuration="5.282055415s" totalImagesPullingTime="3.453677437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:05.277879626 +0000 UTC m=+2158.387722528" watchObservedRunningTime="2026-10-30 12:30:05.282055415 +0000 UTC m=+2158.391898297" Oct 30 12:30:05.319778 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.319722 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4"] Oct 30 12:30:05.323704 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.323675 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7f78d4-6bc954fc54-wj7r4"] Oct 30 12:30:05.731061 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:05.731029 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:05.818601 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:30:05.818563 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:30:06.206437 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:06.206401 4413 generic.go:356] "Generic (PLEG): container finished" podID="d164b031-7ba1-48d2-9890-01e3e941ad3e" containerID="bed5d763a727e55ae983318dd8daec3ed2a6119b6695b92aaf3351b32ca19993" exitCode=60 Oct 30 12:30:06.206946 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:06.206490 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" event={"ID":"d164b031-7ba1-48d2-9890-01e3e941ad3e","Type":"ContainerDied","Data":"bed5d763a727e55ae983318dd8daec3ed2a6119b6695b92aaf3351b32ca19993"} Oct 30 12:30:06.206946 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:06.206829 4413 scope.go:122] "RemoveContainer" containerID="bed5d763a727e55ae983318dd8daec3ed2a6119b6695b92aaf3351b32ca19993" Oct 30 12:30:07.021960 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:07.021926 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="752943e6-f6f7-450b-9e69-0fda21ee5738" path="/var/lib/kubelet/pods/752943e6-f6f7-450b-9e69-0fda21ee5738/volumes" Oct 30 12:30:07.022372 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:07.022355 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8543e221-c552-43fc-a7f4-9d21127db4ef" path="/var/lib/kubelet/pods/8543e221-c552-43fc-a7f4-9d21127db4ef/volumes" Oct 30 12:30:07.216718 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:07.216677 4413 generic.go:356] "Generic (PLEG): container finished" podID="d164b031-7ba1-48d2-9890-01e3e941ad3e" containerID="38e27148fd8f07cedb763deaf3421084860670b0a1b4790c14623624fed5bc75" exitCode=60 Oct 30 12:30:07.217165 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:07.216719 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" event={"ID":"d164b031-7ba1-48d2-9890-01e3e941ad3e","Type":"ContainerDied","Data":"38e27148fd8f07cedb763deaf3421084860670b0a1b4790c14623624fed5bc75"} Oct 30 12:30:07.217165 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:07.216787 4413 scope.go:122] "RemoveContainer" containerID="bed5d763a727e55ae983318dd8daec3ed2a6119b6695b92aaf3351b32ca19993" Oct 30 12:30:07.217165 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:07.217100 4413 scope.go:122] "RemoveContainer" containerID="38e27148fd8f07cedb763deaf3421084860670b0a1b4790c14623624fed5bc75" Oct 30 12:30:07.217533 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:30:07.217511 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz_odh-ai-gateway-infra(d164b031-7ba1-48d2-9890-01e3e941ad3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" podUID="d164b031-7ba1-48d2-9890-01e3e941ad3e" Oct 30 12:30:07.422708 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:07.422671 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:08.225806 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:08.225774 4413 scope.go:122] "RemoveContainer" containerID="38e27148fd8f07cedb763deaf3421084860670b0a1b4790c14623624fed5bc75" Oct 30 12:30:08.226229 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:30:08.226153 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz_odh-ai-gateway-infra(d164b031-7ba1-48d2-9890-01e3e941ad3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" podUID="d164b031-7ba1-48d2-9890-01e3e941ad3e" Oct 30 12:30:09.037169 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.037122 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr"] Oct 30 12:30:09.037907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.037882 4413 cpu_manager.go:414] "containerMap: removing container" podUID="57785552-cd5b-48b5-b4fc-a62d21e76b8d" containerName="istio-proxy" Oct 30 12:30:09.038027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.037909 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57785552-cd5b-48b5-b4fc-a62d21e76b8d" containerName="istio-proxy" Oct 30 12:30:09.038027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.037931 4413 cpu_manager.go:414] "containerMap: removing container" podUID="8543e221-c552-43fc-a7f4-9d21127db4ef" containerName="payload-processing" Oct 30 12:30:09.038027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.037938 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8543e221-c552-43fc-a7f4-9d21127db4ef" containerName="payload-processing" Oct 30 12:30:09.038027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.037987 4413 cpu_manager.go:414] "containerMap: removing container" podUID="752943e6-f6f7-450b-9e69-0fda21ee5738" containerName="payload-pre-processing" Oct 30 12:30:09.038027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.037995 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752943e6-f6f7-450b-9e69-0fda21ee5738" containerName="payload-pre-processing" Oct 30 12:30:09.038348 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.038193 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="752943e6-f6f7-450b-9e69-0fda21ee5738" containerName="payload-pre-processing" Oct 30 12:30:09.038348 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.038220 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="57785552-cd5b-48b5-b4fc-a62d21e76b8d" containerName="istio-proxy" Oct 30 12:30:09.038348 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.038230 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="8543e221-c552-43fc-a7f4-9d21127db4ef" containerName="payload-processing" Oct 30 12:30:09.041931 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.041903 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" Oct 30 12:30:09.071094 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.069960 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-1e9675-serving-cert\"" Oct 30 12:30:09.075266 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.075224 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr"] Oct 30 12:30:09.137761 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.137683 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:09.154334 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.154295 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b83871e-854a-4e91-8857-6c63753834db-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr\" (UID: \"5b83871e-854a-4e91-8857-6c63753834db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" Oct 30 12:30:09.154499 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.154349 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b83871e-854a-4e91-8857-6c63753834db-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr\" (UID: \"5b83871e-854a-4e91-8857-6c63753834db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" Oct 30 12:30:09.154564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.154518 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b83871e-854a-4e91-8857-6c63753834db-metrics-certs\") pod \"maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr\" (UID: \"5b83871e-854a-4e91-8857-6c63753834db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" Oct 30 12:30:09.154597 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.154566 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd9lq\" (UniqueName: \"kubernetes.io/projected/5b83871e-854a-4e91-8857-6c63753834db-kube-api-access-zd9lq\") pod \"maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr\" (UID: \"5b83871e-854a-4e91-8857-6c63753834db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" Oct 30 12:30:09.251547 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.251503 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:09.256321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.255964 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b83871e-854a-4e91-8857-6c63753834db-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr\" (UID: \"5b83871e-854a-4e91-8857-6c63753834db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" Oct 30 12:30:09.256321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.256036 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b83871e-854a-4e91-8857-6c63753834db-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr\" (UID: \"5b83871e-854a-4e91-8857-6c63753834db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" Oct 30 12:30:09.256321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.256145 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b83871e-854a-4e91-8857-6c63753834db-metrics-certs\") pod \"maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr\" (UID: \"5b83871e-854a-4e91-8857-6c63753834db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" Oct 30 12:30:09.256321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.256195 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd9lq\" (UniqueName: \"kubernetes.io/projected/5b83871e-854a-4e91-8857-6c63753834db-kube-api-access-zd9lq\") pod \"maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr\" (UID: \"5b83871e-854a-4e91-8857-6c63753834db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" Oct 30 12:30:09.257702 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.257648 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b83871e-854a-4e91-8857-6c63753834db-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr\" (UID: \"5b83871e-854a-4e91-8857-6c63753834db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" Oct 30 12:30:09.264871 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.264599 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b83871e-854a-4e91-8857-6c63753834db-metrics-certs\") pod \"maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr\" (UID: \"5b83871e-854a-4e91-8857-6c63753834db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" Oct 30 12:30:09.267962 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.267926 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b83871e-854a-4e91-8857-6c63753834db-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr\" (UID: \"5b83871e-854a-4e91-8857-6c63753834db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" Oct 30 12:30:09.279745 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.279701 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd9lq\" (UniqueName: \"kubernetes.io/projected/5b83871e-854a-4e91-8857-6c63753834db-kube-api-access-zd9lq\") pod \"maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr\" (UID: \"5b83871e-854a-4e91-8857-6c63753834db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" Oct 30 12:30:09.355983 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.355895 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" Oct 30 12:30:09.398478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.398443 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:09.598872 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.598833 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:09.665832 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:09.665788 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr"] Oct 30 12:30:09.666426 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:30:09.666402 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b83871e_854a_4e91_8857_6c63753834db.slice/crio-2a668064dd2a2e6409623e219efa8eb52f40fd76b8fd2dffce26d27066d5faa8 WatchSource:0}: Error finding container 2a668064dd2a2e6409623e219efa8eb52f40fd76b8fd2dffce26d27066d5faa8: Status 404 returned error can't find the container with id 2a668064dd2a2e6409623e219efa8eb52f40fd76b8fd2dffce26d27066d5faa8 Oct 30 12:30:10.240030 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:10.239986 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" event={"ID":"5b83871e-854a-4e91-8857-6c63753834db","Type":"ContainerStarted","Data":"265a089eb19dcf450bd61fa1a8a432ac18198d3d49cc8dded812b0aebecc1a92"} Oct 30 12:30:10.240201 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:10.240074 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" event={"ID":"5b83871e-854a-4e91-8857-6c63753834db","Type":"ContainerStarted","Data":"2a668064dd2a2e6409623e219efa8eb52f40fd76b8fd2dffce26d27066d5faa8"} Oct 30 12:30:10.240201 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:10.240095 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" Oct 30 12:30:10.313177 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:10.313105 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" podStartSLOduration=0.832842957 podStartE2EDuration="1.313081445s" totalImagesPullingTime="480.238488ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:10.284126043 +0000 UTC m=+2163.393968924" watchObservedRunningTime="2026-10-30 12:30:10.313081445 +0000 UTC m=+2163.422924327" Oct 30 12:30:11.140669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:11.140641 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:12.928490 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:12.928454 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:13.117297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:13.117220 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:14.368703 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:14.368668 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:14.662096 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:14.662048 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:14.924170 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:14.924076 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:15.093635 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:15.093597 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:15.219057 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:15.218966 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:16.254363 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:16.254317 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" Oct 30 12:30:16.500136 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:16.500103 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:18.404836 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:18.404800 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:18.793735 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:18.793644 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:18.928426 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:18.928384 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:19.961847 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:19.961810 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:20.135123 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:20.135090 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:20.787045 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:20.787006 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:20.909265 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:20.909227 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:21.018088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:21.018061 4413 scope.go:122] "RemoveContainer" containerID="38e27148fd8f07cedb763deaf3421084860670b0a1b4790c14623624fed5bc75" Oct 30 12:30:21.155082 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:21.154957 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t"] Oct 30 12:30:21.166541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:21.166487 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t"] Oct 30 12:30:21.166738 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:21.166620 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t" Oct 30 12:30:21.186941 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:21.186912 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-5ce7c4\"" Oct 30 12:30:21.187074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:21.187056 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-5ce7c4-dockercfg-rkq2j\"" Oct 30 12:30:21.193235 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:21.193198 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06361f15-8bef-4b59-9b23-89c04c941b65-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t\" (UID: \"06361f15-8bef-4b59-9b23-89c04c941b65\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t" Oct 30 12:30:21.193518 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:21.193496 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vr6j\" (UniqueName: \"kubernetes.io/projected/06361f15-8bef-4b59-9b23-89c04c941b65-kube-api-access-6vr6j\") pod \"payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t\" (UID: \"06361f15-8bef-4b59-9b23-89c04c941b65\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t" Oct 30 12:30:21.294398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:21.294314 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06361f15-8bef-4b59-9b23-89c04c941b65-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t\" (UID: \"06361f15-8bef-4b59-9b23-89c04c941b65\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t" Oct 30 12:30:21.294398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:21.294367 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vr6j\" (UniqueName: \"kubernetes.io/projected/06361f15-8bef-4b59-9b23-89c04c941b65-kube-api-access-6vr6j\") pod \"payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t\" (UID: \"06361f15-8bef-4b59-9b23-89c04c941b65\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t" Oct 30 12:30:21.295117 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:21.295095 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06361f15-8bef-4b59-9b23-89c04c941b65-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t\" (UID: \"06361f15-8bef-4b59-9b23-89c04c941b65\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t" Oct 30 12:30:21.323957 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:21.323928 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vr6j\" (UniqueName: \"kubernetes.io/projected/06361f15-8bef-4b59-9b23-89c04c941b65-kube-api-access-6vr6j\") pod \"payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t\" (UID: \"06361f15-8bef-4b59-9b23-89c04c941b65\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t" Oct 30 12:30:21.446242 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:21.446195 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:21.494797 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:21.494767 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t" Oct 30 12:30:21.818940 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:21.818901 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t"] Oct 30 12:30:21.819215 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:30:21.819192 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06361f15_8bef_4b59_9b23_89c04c941b65.slice/crio-ba68fd5562a404c18280db679d84eb5450fa706654cdd781b160f87c3b86d520 WatchSource:0}: Error finding container ba68fd5562a404c18280db679d84eb5450fa706654cdd781b160f87c3b86d520: Status 404 returned error can't find the container with id ba68fd5562a404c18280db679d84eb5450fa706654cdd781b160f87c3b86d520 Oct 30 12:30:22.355958 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:22.355873 4413 generic.go:356] "Generic (PLEG): container finished" podID="d164b031-7ba1-48d2-9890-01e3e941ad3e" containerID="4251fbb6164d48e207fc33ae971968e53901e499a5803d3a5769104c06462429" exitCode=60 Oct 30 12:30:22.355958 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:22.355937 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" event={"ID":"d164b031-7ba1-48d2-9890-01e3e941ad3e","Type":"ContainerDied","Data":"4251fbb6164d48e207fc33ae971968e53901e499a5803d3a5769104c06462429"} Oct 30 12:30:22.356509 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:22.355997 4413 scope.go:122] "RemoveContainer" containerID="38e27148fd8f07cedb763deaf3421084860670b0a1b4790c14623624fed5bc75" Oct 30 12:30:22.356509 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:22.356436 4413 scope.go:122] "RemoveContainer" containerID="4251fbb6164d48e207fc33ae971968e53901e499a5803d3a5769104c06462429" Oct 30 12:30:22.356843 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:30:22.356822 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz_odh-ai-gateway-infra(d164b031-7ba1-48d2-9890-01e3e941ad3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" podUID="d164b031-7ba1-48d2-9890-01e3e941ad3e" Oct 30 12:30:22.357957 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:22.357931 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t" event={"ID":"06361f15-8bef-4b59-9b23-89c04c941b65","Type":"ContainerStarted","Data":"69605c775f6d078ac85b27d68db3b83938b3967a7d935f08287f119a8c324998"} Oct 30 12:30:22.358051 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:22.357999 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t" Oct 30 12:30:22.358051 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:22.358021 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t" event={"ID":"06361f15-8bef-4b59-9b23-89c04c941b65","Type":"ContainerStarted","Data":"ba68fd5562a404c18280db679d84eb5450fa706654cdd781b160f87c3b86d520"} Oct 30 12:30:22.439169 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:22.439104 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t" podStartSLOduration=1.4390892530000001 podStartE2EDuration="1.439089253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:22.438827618 +0000 UTC m=+2175.548670498" watchObservedRunningTime="2026-10-30 12:30:22.439089253 +0000 UTC m=+2175.548932134" Oct 30 12:30:23.411195 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:23.411160 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz"] Oct 30 12:30:23.568328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:23.568297 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" Oct 30 12:30:23.631517 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:23.631483 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d164b031-7ba1-48d2-9890-01e3e941ad3e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d164b031-7ba1-48d2-9890-01e3e941ad3e-maas-api-service-ca\") pod \"d164b031-7ba1-48d2-9890-01e3e941ad3e\" (UID: \"d164b031-7ba1-48d2-9890-01e3e941ad3e\") " Oct 30 12:30:23.631697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:23.631593 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d164b031-7ba1-48d2-9890-01e3e941ad3e-kube-api-access-xpwqj\" (UniqueName: \"kubernetes.io/projected/d164b031-7ba1-48d2-9890-01e3e941ad3e-kube-api-access-xpwqj\") pod \"d164b031-7ba1-48d2-9890-01e3e941ad3e\" (UID: \"d164b031-7ba1-48d2-9890-01e3e941ad3e\") " Oct 30 12:30:23.631911 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:23.631884 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d164b031-7ba1-48d2-9890-01e3e941ad3e-maas-api-service-ca" pod "d164b031-7ba1-48d2-9890-01e3e941ad3e" (UID: "d164b031-7ba1-48d2-9890-01e3e941ad3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:23.632166 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:23.632146 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d164b031-7ba1-48d2-9890-01e3e941ad3e-maas-api-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:23.634359 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:23.634322 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d164b031-7ba1-48d2-9890-01e3e941ad3e-kube-api-access-xpwqj" pod "d164b031-7ba1-48d2-9890-01e3e941ad3e" (UID: "d164b031-7ba1-48d2-9890-01e3e941ad3e"). InnerVolumeSpecName "kube-api-access-xpwqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:23.642927 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:23.642880 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:23.733808 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:23.733725 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpwqj\" (UniqueName: \"kubernetes.io/projected/d164b031-7ba1-48d2-9890-01e3e941ad3e-kube-api-access-xpwqj\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:24.373976 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:24.373947 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" Oct 30 12:30:24.374301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:24.373985 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz" event={"ID":"d164b031-7ba1-48d2-9890-01e3e941ad3e","Type":"ContainerDied","Data":"57704dafa789170a913030fb14f4a8b5ed47766fde2c81eb500ba119b0bc2356"} Oct 30 12:30:24.374301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:24.374051 4413 scope.go:122] "RemoveContainer" containerID="4251fbb6164d48e207fc33ae971968e53901e499a5803d3a5769104c06462429" Oct 30 12:30:24.426516 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:24.426485 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz"] Oct 30 12:30:24.437936 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:24.437903 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7eee3b-29889390-8pxzz"] Oct 30 12:30:25.022076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:25.022044 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d164b031-7ba1-48d2-9890-01e3e941ad3e" path="/var/lib/kubelet/pods/d164b031-7ba1-48d2-9890-01e3e941ad3e/volumes" Oct 30 12:30:25.279967 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:25.279888 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:26.876228 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:26.876192 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:27.782573 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:27.782540 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:27.900085 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:27.900053 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:28.836000 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:28.835959 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:29.429973 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:29.429933 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:29.545112 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:29.545058 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:29.662750 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:29.662712 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:30.070280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:30.070214 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:31.420476 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:31.420433 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:33.181072 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:33.180950 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:33.369866 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:33.369838 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t" Oct 30 12:30:35.014686 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:35.014652 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:35.816071 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:30:35.816030 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:30:36.423828 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:36.423787 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:36.593053 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:36.593017 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:36.703587 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:36.703505 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:37.307770 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:37.307724 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:37.421456 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:37.421423 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:37.831821 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:37.831776 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:37.942500 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:37.942472 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:38.053820 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:38.053788 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:38.192177 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:38.192142 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:39.249699 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:39.249653 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:39.362594 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:39.362557 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:40.937586 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:40.937550 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:42.709322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:42.709287 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:43.033515 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:43.033432 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:44.320334 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:44.320303 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:44.431173 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:44.431139 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:46.168090 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:46.168047 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:47.973557 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:47.973521 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh"] Oct 30 12:30:47.974116 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:47.973784 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" podUID="a1a91278-7714-4ff4-a35a-5f573a09497a" containerName="maas-api" containerID="cri-o://9c0aa062c5521dfb1fd86bd139f16ce992339d622d3de0134fcf9527d290f1d0" gracePeriod=30 Oct 30 12:30:48.066970 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.066937 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:48.145779 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.145739 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5"] Oct 30 12:30:48.146056 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.146027 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5" podUID="d8af5dd5-cd57-4892-86dd-54f2b3b203a0" containerName="payload-pre-processing" containerID="cri-o://36c2f269f5df316fe2c161d25e3bc1ae45b1b215b3a7539f351effc235feaa48" gracePeriod=30 Oct 30 12:30:48.215543 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.215080 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:48.303976 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.303951 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:30:48.325074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.325031 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:48.411379 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.411354 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a91278-7714-4ff4-a35a-5f573a09497a-kube-api-access-w5c29\" (UniqueName: \"kubernetes.io/projected/a1a91278-7714-4ff4-a35a-5f573a09497a-kube-api-access-w5c29\") pod \"a1a91278-7714-4ff4-a35a-5f573a09497a\" (UID: \"a1a91278-7714-4ff4-a35a-5f573a09497a\") " Oct 30 12:30:48.411509 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.411476 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a91278-7714-4ff4-a35a-5f573a09497a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a91278-7714-4ff4-a35a-5f573a09497a-metrics-certs\") pod \"a1a91278-7714-4ff4-a35a-5f573a09497a\" (UID: \"a1a91278-7714-4ff4-a35a-5f573a09497a\") " Oct 30 12:30:48.411601 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.411587 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a91278-7714-4ff4-a35a-5f573a09497a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a91278-7714-4ff4-a35a-5f573a09497a-maas-api-tls\") pod \"a1a91278-7714-4ff4-a35a-5f573a09497a\" (UID: \"a1a91278-7714-4ff4-a35a-5f573a09497a\") " Oct 30 12:30:48.411673 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.411650 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a91278-7714-4ff4-a35a-5f573a09497a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a91278-7714-4ff4-a35a-5f573a09497a-openshift-service-ca\") pod \"a1a91278-7714-4ff4-a35a-5f573a09497a\" (UID: \"a1a91278-7714-4ff4-a35a-5f573a09497a\") " Oct 30 12:30:48.412028 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.412003 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a91278-7714-4ff4-a35a-5f573a09497a-openshift-service-ca" pod "a1a91278-7714-4ff4-a35a-5f573a09497a" (UID: "a1a91278-7714-4ff4-a35a-5f573a09497a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:48.412434 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.412406 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a91278-7714-4ff4-a35a-5f573a09497a-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:48.413737 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.413711 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a91278-7714-4ff4-a35a-5f573a09497a-kube-api-access-w5c29" pod "a1a91278-7714-4ff4-a35a-5f573a09497a" (UID: "a1a91278-7714-4ff4-a35a-5f573a09497a"). InnerVolumeSpecName "kube-api-access-w5c29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:48.413819 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.413751 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a91278-7714-4ff4-a35a-5f573a09497a-metrics-certs" pod "a1a91278-7714-4ff4-a35a-5f573a09497a" (UID: "a1a91278-7714-4ff4-a35a-5f573a09497a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:30:48.413867 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.413826 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a91278-7714-4ff4-a35a-5f573a09497a-maas-api-tls" pod "a1a91278-7714-4ff4-a35a-5f573a09497a" (UID: "a1a91278-7714-4ff4-a35a-5f573a09497a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:30:48.418973 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.418955 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5" Oct 30 12:30:48.537366 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.524710 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8af5dd5-cd57-4892-86dd-54f2b3b203a0-kube-api-access-gnlsg\" (UniqueName: \"kubernetes.io/projected/d8af5dd5-cd57-4892-86dd-54f2b3b203a0-kube-api-access-gnlsg\") pod \"d8af5dd5-cd57-4892-86dd-54f2b3b203a0\" (UID: \"d8af5dd5-cd57-4892-86dd-54f2b3b203a0\") " Oct 30 12:30:48.537366 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.524781 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8af5dd5-cd57-4892-86dd-54f2b3b203a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8af5dd5-cd57-4892-86dd-54f2b3b203a0-plugins-config-volume\") pod \"d8af5dd5-cd57-4892-86dd-54f2b3b203a0\" (UID: \"d8af5dd5-cd57-4892-86dd-54f2b3b203a0\") " Oct 30 12:30:48.537366 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.525385 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a91278-7714-4ff4-a35a-5f573a09497a-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:48.537366 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.525406 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5c29\" (UniqueName: \"kubernetes.io/projected/a1a91278-7714-4ff4-a35a-5f573a09497a-kube-api-access-w5c29\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:48.537366 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.525421 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a91278-7714-4ff4-a35a-5f573a09497a-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:48.537366 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.525845 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8af5dd5-cd57-4892-86dd-54f2b3b203a0-plugins-config-volume" pod "d8af5dd5-cd57-4892-86dd-54f2b3b203a0" (UID: "d8af5dd5-cd57-4892-86dd-54f2b3b203a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:48.553941 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.553884 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8af5dd5-cd57-4892-86dd-54f2b3b203a0-kube-api-access-gnlsg" pod "d8af5dd5-cd57-4892-86dd-54f2b3b203a0" (UID: "d8af5dd5-cd57-4892-86dd-54f2b3b203a0"). InnerVolumeSpecName "kube-api-access-gnlsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:48.568994 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.568955 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:48.569331 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.569095 4413 generic.go:356] "Generic (PLEG): container finished" podID="d8af5dd5-cd57-4892-86dd-54f2b3b203a0" containerID="36c2f269f5df316fe2c161d25e3bc1ae45b1b215b3a7539f351effc235feaa48" exitCode=0 Oct 30 12:30:48.569715 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.569126 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5" event={"ID":"d8af5dd5-cd57-4892-86dd-54f2b3b203a0","Type":"ContainerDied","Data":"36c2f269f5df316fe2c161d25e3bc1ae45b1b215b3a7539f351effc235feaa48"} Oct 30 12:30:48.569907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.569888 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5" event={"ID":"d8af5dd5-cd57-4892-86dd-54f2b3b203a0","Type":"ContainerDied","Data":"7f61bbd549232c9b86c783f292cd857048c45a58f23269d7a503a68f0bc66387"} Oct 30 12:30:48.570050 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.570037 4413 scope.go:122] "RemoveContainer" containerID="36c2f269f5df316fe2c161d25e3bc1ae45b1b215b3a7539f351effc235feaa48" Oct 30 12:30:48.570240 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.569204 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5" Oct 30 12:30:48.578013 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.577968 4413 generic.go:356] "Generic (PLEG): container finished" podID="a1a91278-7714-4ff4-a35a-5f573a09497a" containerID="9c0aa062c5521dfb1fd86bd139f16ce992339d622d3de0134fcf9527d290f1d0" exitCode=0 Oct 30 12:30:48.578139 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.578066 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" event={"ID":"a1a91278-7714-4ff4-a35a-5f573a09497a","Type":"ContainerDied","Data":"9c0aa062c5521dfb1fd86bd139f16ce992339d622d3de0134fcf9527d290f1d0"} Oct 30 12:30:48.578139 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.578110 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" event={"ID":"a1a91278-7714-4ff4-a35a-5f573a09497a","Type":"ContainerDied","Data":"58243f44433c1e7edfdf0c93d9edb572fdbf9588b1705f3bedf1374be03bfab6"} Oct 30 12:30:48.578324 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.578187 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh" Oct 30 12:30:48.596077 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.596054 4413 scope.go:122] "RemoveContainer" containerID="36c2f269f5df316fe2c161d25e3bc1ae45b1b215b3a7539f351effc235feaa48" Oct 30 12:30:48.596522 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:30:48.596492 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c2f269f5df316fe2c161d25e3bc1ae45b1b215b3a7539f351effc235feaa48\": container with ID starting with 36c2f269f5df316fe2c161d25e3bc1ae45b1b215b3a7539f351effc235feaa48 not found: ID does not exist" containerID="36c2f269f5df316fe2c161d25e3bc1ae45b1b215b3a7539f351effc235feaa48" Oct 30 12:30:48.596635 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.596536 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c2f269f5df316fe2c161d25e3bc1ae45b1b215b3a7539f351effc235feaa48"} err="failed to get container status \"36c2f269f5df316fe2c161d25e3bc1ae45b1b215b3a7539f351effc235feaa48\": rpc error: code = NotFound desc = could not find container \"36c2f269f5df316fe2c161d25e3bc1ae45b1b215b3a7539f351effc235feaa48\": container with ID starting with 36c2f269f5df316fe2c161d25e3bc1ae45b1b215b3a7539f351effc235feaa48 not found: ID does not exist" Oct 30 12:30:48.596635 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.596580 4413 scope.go:122] "RemoveContainer" containerID="9c0aa062c5521dfb1fd86bd139f16ce992339d622d3de0134fcf9527d290f1d0" Oct 30 12:30:48.607864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.607844 4413 scope.go:122] "RemoveContainer" containerID="9c0aa062c5521dfb1fd86bd139f16ce992339d622d3de0134fcf9527d290f1d0" Oct 30 12:30:48.608146 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:30:48.608124 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c0aa062c5521dfb1fd86bd139f16ce992339d622d3de0134fcf9527d290f1d0\": container with ID starting with 9c0aa062c5521dfb1fd86bd139f16ce992339d622d3de0134fcf9527d290f1d0 not found: ID does not exist" containerID="9c0aa062c5521dfb1fd86bd139f16ce992339d622d3de0134fcf9527d290f1d0" Oct 30 12:30:48.608196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.608159 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c0aa062c5521dfb1fd86bd139f16ce992339d622d3de0134fcf9527d290f1d0"} err="failed to get container status \"9c0aa062c5521dfb1fd86bd139f16ce992339d622d3de0134fcf9527d290f1d0\": rpc error: code = NotFound desc = could not find container \"9c0aa062c5521dfb1fd86bd139f16ce992339d622d3de0134fcf9527d290f1d0\": container with ID starting with 9c0aa062c5521dfb1fd86bd139f16ce992339d622d3de0134fcf9527d290f1d0 not found: ID does not exist" Oct 30 12:30:48.626610 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.626575 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnlsg\" (UniqueName: \"kubernetes.io/projected/d8af5dd5-cd57-4892-86dd-54f2b3b203a0-kube-api-access-gnlsg\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:48.626610 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.626611 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8af5dd5-cd57-4892-86dd-54f2b3b203a0-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:48.627826 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.627799 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5"] Oct 30 12:30:48.637027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.637004 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7eee3b-58cbc585b7-g5hs5"] Oct 30 12:30:48.653673 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:30:48.653636 4413 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8af5dd5_cd57_4892_86dd_54f2b3b203a0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a91278_7714_4ff4_a35a_5f573a09497a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a91278_7714_4ff4_a35a_5f573a09497a.slice/crio-58243f44433c1e7edfdf0c93d9edb572fdbf9588b1705f3bedf1374be03bfab6\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8af5dd5_cd57_4892_86dd_54f2b3b203a0.slice/crio-7f61bbd549232c9b86c783f292cd857048c45a58f23269d7a503a68f0bc66387\": RecentStats: unable to find data in memory cache]" Oct 30 12:30:48.684964 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.684938 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh"] Oct 30 12:30:48.689028 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.689003 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7eee3b-5b68d84c66-mq5kh"] Oct 30 12:30:48.730474 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.730436 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:48.847747 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:48.847646 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:49.021164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:49.021092 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a91278-7714-4ff4-a35a-5f573a09497a" path="/var/lib/kubelet/pods/a1a91278-7714-4ff4-a35a-5f573a09497a/volumes" Oct 30 12:30:49.021547 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:49.021512 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8af5dd5-cd57-4892-86dd-54f2b3b203a0" path="/var/lib/kubelet/pods/d8af5dd5-cd57-4892-86dd-54f2b3b203a0/volumes" Oct 30 12:30:49.233193 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:49.233158 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn"] Oct 30 12:30:49.289919 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:49.289885 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:49.403469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:49.403436 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:49.517969 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:49.517877 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:50.551004 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:50.550949 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:52.168855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:52.168824 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h"] Oct 30 12:30:52.200479 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:52.200439 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:52.201201 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:52.201183 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96"] Oct 30 12:30:52.316657 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:52.316630 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:52.430759 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:52.430667 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:52.827135 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:52.827040 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s"] Oct 30 12:30:52.827662 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:52.827610 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" podUID="799fe64f-b3d4-43bf-b7ae-04833737b666" containerName="discovery" containerID="cri-o://fb8c919024c0a744264bcb9c5e8b2ee24cea9c447d17b00e77b4c6153b1a655c" gracePeriod=30 Oct 30 12:30:53.116902 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.116878 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:30:53.206399 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.206370 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/799fe64f-b3d4-43bf-b7ae-04833737b666-kube-api-access-hc75m\" (UniqueName: \"kubernetes.io/projected/799fe64f-b3d4-43bf-b7ae-04833737b666-kube-api-access-hc75m\") pod \"799fe64f-b3d4-43bf-b7ae-04833737b666\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " Oct 30 12:30:53.206900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.206456 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-cacerts\" (UniqueName: \"kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-cacerts\") pod \"799fe64f-b3d4-43bf-b7ae-04833737b666\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " Oct 30 12:30:53.206900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.206613 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-csr-ca-configmap\") pod \"799fe64f-b3d4-43bf-b7ae-04833737b666\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " Oct 30 12:30:53.206900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.206687 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-kubeconfig\") pod \"799fe64f-b3d4-43bf-b7ae-04833737b666\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " Oct 30 12:30:53.206900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.206783 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-token\" (UniqueName: \"kubernetes.io/projected/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-token\") pod \"799fe64f-b3d4-43bf-b7ae-04833737b666\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " Oct 30 12:30:53.206900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.206817 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799fe64f-b3d4-43bf-b7ae-04833737b666-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/799fe64f-b3d4-43bf-b7ae-04833737b666-local-certs\") pod \"799fe64f-b3d4-43bf-b7ae-04833737b666\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " Oct 30 12:30:53.207359 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.206906 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-csr-dns-cert\") pod \"799fe64f-b3d4-43bf-b7ae-04833737b666\" (UID: \"799fe64f-b3d4-43bf-b7ae-04833737b666\") " Oct 30 12:30:53.207359 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.206976 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-csr-ca-configmap" pod "799fe64f-b3d4-43bf-b7ae-04833737b666" (UID: "799fe64f-b3d4-43bf-b7ae-04833737b666"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:53.207755 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.207729 4413 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-csr-ca-configmap\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:53.209741 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.209713 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799fe64f-b3d4-43bf-b7ae-04833737b666-local-certs" pod "799fe64f-b3d4-43bf-b7ae-04833737b666" (UID: "799fe64f-b3d4-43bf-b7ae-04833737b666"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:30:53.209841 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.209756 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-cacerts" pod "799fe64f-b3d4-43bf-b7ae-04833737b666" (UID: "799fe64f-b3d4-43bf-b7ae-04833737b666"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:30:53.209841 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.209774 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-token" pod "799fe64f-b3d4-43bf-b7ae-04833737b666" (UID: "799fe64f-b3d4-43bf-b7ae-04833737b666"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:53.209841 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.209781 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-kubeconfig" pod "799fe64f-b3d4-43bf-b7ae-04833737b666" (UID: "799fe64f-b3d4-43bf-b7ae-04833737b666"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:30:53.210012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.209994 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799fe64f-b3d4-43bf-b7ae-04833737b666-kube-api-access-hc75m" pod "799fe64f-b3d4-43bf-b7ae-04833737b666" (UID: "799fe64f-b3d4-43bf-b7ae-04833737b666"). InnerVolumeSpecName "kube-api-access-hc75m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:53.210705 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.210682 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-csr-dns-cert" pod "799fe64f-b3d4-43bf-b7ae-04833737b666" (UID: "799fe64f-b3d4-43bf-b7ae-04833737b666"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:30:53.308740 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.308707 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc75m\" (UniqueName: \"kubernetes.io/projected/799fe64f-b3d4-43bf-b7ae-04833737b666-kube-api-access-hc75m\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:53.308740 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.308735 4413 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-cacerts\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:53.308740 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.308745 4413 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-kubeconfig\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:53.308984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.308754 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:53.308984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.308762 4413 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/799fe64f-b3d4-43bf-b7ae-04833737b666-local-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:53.308984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.308769 4413 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/799fe64f-b3d4-43bf-b7ae-04833737b666-istio-csr-dns-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:53.627257 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.627220 4413 generic.go:356] "Generic (PLEG): container finished" podID="799fe64f-b3d4-43bf-b7ae-04833737b666" containerID="fb8c919024c0a744264bcb9c5e8b2ee24cea9c447d17b00e77b4c6153b1a655c" exitCode=0 Oct 30 12:30:53.627448 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.627298 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" event={"ID":"799fe64f-b3d4-43bf-b7ae-04833737b666","Type":"ContainerDied","Data":"fb8c919024c0a744264bcb9c5e8b2ee24cea9c447d17b00e77b4c6153b1a655c"} Oct 30 12:30:53.627448 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.627320 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" Oct 30 12:30:53.627448 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.627366 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s" event={"ID":"799fe64f-b3d4-43bf-b7ae-04833737b666","Type":"ContainerDied","Data":"da0138d52bc45df7441a3bbc877092f07b4ff349d9c5920f9b3c24bc14c6c7ce"} Oct 30 12:30:53.627448 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.627388 4413 scope.go:122] "RemoveContainer" containerID="fb8c919024c0a744264bcb9c5e8b2ee24cea9c447d17b00e77b4c6153b1a655c" Oct 30 12:30:53.638762 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.638740 4413 scope.go:122] "RemoveContainer" containerID="fb8c919024c0a744264bcb9c5e8b2ee24cea9c447d17b00e77b4c6153b1a655c" Oct 30 12:30:53.639078 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:30:53.639059 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb8c919024c0a744264bcb9c5e8b2ee24cea9c447d17b00e77b4c6153b1a655c\": container with ID starting with fb8c919024c0a744264bcb9c5e8b2ee24cea9c447d17b00e77b4c6153b1a655c not found: ID does not exist" containerID="fb8c919024c0a744264bcb9c5e8b2ee24cea9c447d17b00e77b4c6153b1a655c" Oct 30 12:30:53.639144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.639087 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb8c919024c0a744264bcb9c5e8b2ee24cea9c447d17b00e77b4c6153b1a655c"} err="failed to get container status \"fb8c919024c0a744264bcb9c5e8b2ee24cea9c447d17b00e77b4c6153b1a655c\": rpc error: code = NotFound desc = could not find container \"fb8c919024c0a744264bcb9c5e8b2ee24cea9c447d17b00e77b4c6153b1a655c\": container with ID starting with fb8c919024c0a744264bcb9c5e8b2ee24cea9c447d17b00e77b4c6153b1a655c not found: ID does not exist" Oct 30 12:30:53.687996 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.687963 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s"] Oct 30 12:30:53.699989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:53.699954 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-82z8s"] Oct 30 12:30:54.234207 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:54.234166 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn" podUID="2d20b6be-6972-426c-afff-bb5ae5f93a6e" containerName="payload-processing" containerID="cri-o://772557a91b375d2a2bdf88e7a17364d8ff66a40ad301ceeba30390d8f824572d" gracePeriod=25 Oct 30 12:30:54.452500 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:54.452454 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:54.502218 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:54.502131 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn" Oct 30 12:30:54.627190 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:54.627157 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d20b6be-6972-426c-afff-bb5ae5f93a6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d20b6be-6972-426c-afff-bb5ae5f93a6e-plugins-config-volume\") pod \"2d20b6be-6972-426c-afff-bb5ae5f93a6e\" (UID: \"2d20b6be-6972-426c-afff-bb5ae5f93a6e\") " Oct 30 12:30:54.627462 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:54.627445 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d20b6be-6972-426c-afff-bb5ae5f93a6e-kube-api-access-59rqs\" (UniqueName: \"kubernetes.io/projected/2d20b6be-6972-426c-afff-bb5ae5f93a6e-kube-api-access-59rqs\") pod \"2d20b6be-6972-426c-afff-bb5ae5f93a6e\" (UID: \"2d20b6be-6972-426c-afff-bb5ae5f93a6e\") " Oct 30 12:30:54.627589 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:54.627565 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d20b6be-6972-426c-afff-bb5ae5f93a6e-plugins-config-volume" pod "2d20b6be-6972-426c-afff-bb5ae5f93a6e" (UID: "2d20b6be-6972-426c-afff-bb5ae5f93a6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:54.627951 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:54.627933 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d20b6be-6972-426c-afff-bb5ae5f93a6e-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:54.629931 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:54.629911 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d20b6be-6972-426c-afff-bb5ae5f93a6e-kube-api-access-59rqs" pod "2d20b6be-6972-426c-afff-bb5ae5f93a6e" (UID: "2d20b6be-6972-426c-afff-bb5ae5f93a6e"). InnerVolumeSpecName "kube-api-access-59rqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:54.635739 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:54.635714 4413 generic.go:356] "Generic (PLEG): container finished" podID="2d20b6be-6972-426c-afff-bb5ae5f93a6e" containerID="772557a91b375d2a2bdf88e7a17364d8ff66a40ad301ceeba30390d8f824572d" exitCode=0 Oct 30 12:30:54.635837 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:54.635763 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn" Oct 30 12:30:54.635837 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:54.635772 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn" event={"ID":"2d20b6be-6972-426c-afff-bb5ae5f93a6e","Type":"ContainerDied","Data":"772557a91b375d2a2bdf88e7a17364d8ff66a40ad301ceeba30390d8f824572d"} Oct 30 12:30:54.635837 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:54.635821 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn" event={"ID":"2d20b6be-6972-426c-afff-bb5ae5f93a6e","Type":"ContainerDied","Data":"b480c230228c5110ef276c71e61b8feefbe05ec24f2ea3601af8c7943aa5121c"} Oct 30 12:30:54.635997 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:54.635846 4413 scope.go:122] "RemoveContainer" containerID="772557a91b375d2a2bdf88e7a17364d8ff66a40ad301ceeba30390d8f824572d" Oct 30 12:30:54.647854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:54.647832 4413 scope.go:122] "RemoveContainer" containerID="772557a91b375d2a2bdf88e7a17364d8ff66a40ad301ceeba30390d8f824572d" Oct 30 12:30:54.648230 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:30:54.648204 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"772557a91b375d2a2bdf88e7a17364d8ff66a40ad301ceeba30390d8f824572d\": container with ID starting with 772557a91b375d2a2bdf88e7a17364d8ff66a40ad301ceeba30390d8f824572d not found: ID does not exist" containerID="772557a91b375d2a2bdf88e7a17364d8ff66a40ad301ceeba30390d8f824572d" Oct 30 12:30:54.648351 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:54.648240 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"772557a91b375d2a2bdf88e7a17364d8ff66a40ad301ceeba30390d8f824572d"} err="failed to get container status \"772557a91b375d2a2bdf88e7a17364d8ff66a40ad301ceeba30390d8f824572d\": rpc error: code = NotFound desc = could not find container \"772557a91b375d2a2bdf88e7a17364d8ff66a40ad301ceeba30390d8f824572d\": container with ID starting with 772557a91b375d2a2bdf88e7a17364d8ff66a40ad301ceeba30390d8f824572d not found: ID does not exist" Oct 30 12:30:54.687622 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:54.687593 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn"] Oct 30 12:30:54.694324 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:54.694299 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ec236e-5cdfd995f8-qrlkn"] Oct 30 12:30:54.729807 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:54.729764 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59rqs\" (UniqueName: \"kubernetes.io/projected/2d20b6be-6972-426c-afff-bb5ae5f93a6e-kube-api-access-59rqs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:55.024186 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:55.024154 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d20b6be-6972-426c-afff-bb5ae5f93a6e" path="/var/lib/kubelet/pods/2d20b6be-6972-426c-afff-bb5ae5f93a6e/volumes" Oct 30 12:30:55.024684 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:55.024670 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="799fe64f-b3d4-43bf-b7ae-04833737b666" path="/var/lib/kubelet/pods/799fe64f-b3d4-43bf-b7ae-04833737b666/volumes" Oct 30 12:30:55.981958 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:55.981924 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:56.107482 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:56.107454 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:56.220388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:56.220354 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:56.355783 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:56.355706 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:57.169590 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.169550 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h" podUID="56ad34b9-579c-489a-bef6-f595989c2e86" containerName="payload-processing" containerID="cri-o://1d80f1b23f02fb8e4751d51910c134962a0d1998e6f922f1a756165ab95b2a83" gracePeriod=25 Oct 30 12:30:57.202155 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.202102 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96" podUID="719def6b-4f47-4d2d-8665-a7fa3af2311f" containerName="payload-pre-processing" containerID="cri-o://2f008856d0d32d8dd34f4737cf68642d203eaab83ec4810e42d61bfdd12b0ae7" gracePeriod=25 Oct 30 12:30:57.553530 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.553506 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h" Oct 30 12:30:57.557175 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.557156 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96" Oct 30 12:30:57.659716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.659683 4413 generic.go:356] "Generic (PLEG): container finished" podID="719def6b-4f47-4d2d-8665-a7fa3af2311f" containerID="2f008856d0d32d8dd34f4737cf68642d203eaab83ec4810e42d61bfdd12b0ae7" exitCode=0 Oct 30 12:30:57.659911 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.659751 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96" Oct 30 12:30:57.659911 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.659774 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96" event={"ID":"719def6b-4f47-4d2d-8665-a7fa3af2311f","Type":"ContainerDied","Data":"2f008856d0d32d8dd34f4737cf68642d203eaab83ec4810e42d61bfdd12b0ae7"} Oct 30 12:30:57.659911 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.659833 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96" event={"ID":"719def6b-4f47-4d2d-8665-a7fa3af2311f","Type":"ContainerDied","Data":"30a5c2e546bac017d8560e2019102748bc4a2ab1aee29fb1ab036f4504f913b6"} Oct 30 12:30:57.659911 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.659861 4413 scope.go:122] "RemoveContainer" containerID="2f008856d0d32d8dd34f4737cf68642d203eaab83ec4810e42d61bfdd12b0ae7" Oct 30 12:30:57.661419 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.661389 4413 generic.go:356] "Generic (PLEG): container finished" podID="56ad34b9-579c-489a-bef6-f595989c2e86" containerID="1d80f1b23f02fb8e4751d51910c134962a0d1998e6f922f1a756165ab95b2a83" exitCode=0 Oct 30 12:30:57.661530 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.661482 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h" event={"ID":"56ad34b9-579c-489a-bef6-f595989c2e86","Type":"ContainerDied","Data":"1d80f1b23f02fb8e4751d51910c134962a0d1998e6f922f1a756165ab95b2a83"} Oct 30 12:30:57.661706 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.661670 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h" event={"ID":"56ad34b9-579c-489a-bef6-f595989c2e86","Type":"ContainerDied","Data":"0e9bd708a4493007c174d42933be7d0cd05845ab69afbe8bdb31dd92f1320755"} Oct 30 12:30:57.661778 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.661766 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h" Oct 30 12:30:57.668538 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.668510 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/719def6b-4f47-4d2d-8665-a7fa3af2311f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719def6b-4f47-4d2d-8665-a7fa3af2311f-plugins-config-volume\") pod \"719def6b-4f47-4d2d-8665-a7fa3af2311f\" (UID: \"719def6b-4f47-4d2d-8665-a7fa3af2311f\") " Oct 30 12:30:57.668659 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.668642 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ad34b9-579c-489a-bef6-f595989c2e86-kube-api-access-ffw8j\" (UniqueName: \"kubernetes.io/projected/56ad34b9-579c-489a-bef6-f595989c2e86-kube-api-access-ffw8j\") pod \"56ad34b9-579c-489a-bef6-f595989c2e86\" (UID: \"56ad34b9-579c-489a-bef6-f595989c2e86\") " Oct 30 12:30:57.668794 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.668779 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/719def6b-4f47-4d2d-8665-a7fa3af2311f-kube-api-access-5lb8s\" (UniqueName: \"kubernetes.io/projected/719def6b-4f47-4d2d-8665-a7fa3af2311f-kube-api-access-5lb8s\") pod \"719def6b-4f47-4d2d-8665-a7fa3af2311f\" (UID: \"719def6b-4f47-4d2d-8665-a7fa3af2311f\") " Oct 30 12:30:57.668847 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.668815 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56ad34b9-579c-489a-bef6-f595989c2e86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ad34b9-579c-489a-bef6-f595989c2e86-plugins-config-volume\") pod \"56ad34b9-579c-489a-bef6-f595989c2e86\" (UID: \"56ad34b9-579c-489a-bef6-f595989c2e86\") " Oct 30 12:30:57.668945 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.668926 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719def6b-4f47-4d2d-8665-a7fa3af2311f-plugins-config-volume" pod "719def6b-4f47-4d2d-8665-a7fa3af2311f" (UID: "719def6b-4f47-4d2d-8665-a7fa3af2311f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:57.669266 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.669221 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ad34b9-579c-489a-bef6-f595989c2e86-plugins-config-volume" pod "56ad34b9-579c-489a-bef6-f595989c2e86" (UID: "56ad34b9-579c-489a-bef6-f595989c2e86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:30:57.669381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.669366 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ad34b9-579c-489a-bef6-f595989c2e86-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:57.669559 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.669383 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719def6b-4f47-4d2d-8665-a7fa3af2311f-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:57.670314 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.670292 4413 scope.go:122] "RemoveContainer" containerID="2f008856d0d32d8dd34f4737cf68642d203eaab83ec4810e42d61bfdd12b0ae7" Oct 30 12:30:57.670651 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:30:57.670628 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f008856d0d32d8dd34f4737cf68642d203eaab83ec4810e42d61bfdd12b0ae7\": container with ID starting with 2f008856d0d32d8dd34f4737cf68642d203eaab83ec4810e42d61bfdd12b0ae7 not found: ID does not exist" containerID="2f008856d0d32d8dd34f4737cf68642d203eaab83ec4810e42d61bfdd12b0ae7" Oct 30 12:30:57.670724 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.670659 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f008856d0d32d8dd34f4737cf68642d203eaab83ec4810e42d61bfdd12b0ae7"} err="failed to get container status \"2f008856d0d32d8dd34f4737cf68642d203eaab83ec4810e42d61bfdd12b0ae7\": rpc error: code = NotFound desc = could not find container \"2f008856d0d32d8dd34f4737cf68642d203eaab83ec4810e42d61bfdd12b0ae7\": container with ID starting with 2f008856d0d32d8dd34f4737cf68642d203eaab83ec4810e42d61bfdd12b0ae7 not found: ID does not exist" Oct 30 12:30:57.670724 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.670677 4413 scope.go:122] "RemoveContainer" containerID="1d80f1b23f02fb8e4751d51910c134962a0d1998e6f922f1a756165ab95b2a83" Oct 30 12:30:57.671221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.671194 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ad34b9-579c-489a-bef6-f595989c2e86-kube-api-access-ffw8j" pod "56ad34b9-579c-489a-bef6-f595989c2e86" (UID: "56ad34b9-579c-489a-bef6-f595989c2e86"). InnerVolumeSpecName "kube-api-access-ffw8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:57.671415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.671228 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719def6b-4f47-4d2d-8665-a7fa3af2311f-kube-api-access-5lb8s" pod "719def6b-4f47-4d2d-8665-a7fa3af2311f" (UID: "719def6b-4f47-4d2d-8665-a7fa3af2311f"). InnerVolumeSpecName "kube-api-access-5lb8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:30:57.680403 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.680380 4413 scope.go:122] "RemoveContainer" containerID="1d80f1b23f02fb8e4751d51910c134962a0d1998e6f922f1a756165ab95b2a83" Oct 30 12:30:57.680741 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:30:57.680716 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d80f1b23f02fb8e4751d51910c134962a0d1998e6f922f1a756165ab95b2a83\": container with ID starting with 1d80f1b23f02fb8e4751d51910c134962a0d1998e6f922f1a756165ab95b2a83 not found: ID does not exist" containerID="1d80f1b23f02fb8e4751d51910c134962a0d1998e6f922f1a756165ab95b2a83" Oct 30 12:30:57.680803 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.680754 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d80f1b23f02fb8e4751d51910c134962a0d1998e6f922f1a756165ab95b2a83"} err="failed to get container status \"1d80f1b23f02fb8e4751d51910c134962a0d1998e6f922f1a756165ab95b2a83\": rpc error: code = NotFound desc = could not find container \"1d80f1b23f02fb8e4751d51910c134962a0d1998e6f922f1a756165ab95b2a83\": container with ID starting with 1d80f1b23f02fb8e4751d51910c134962a0d1998e6f922f1a756165ab95b2a83 not found: ID does not exist" Oct 30 12:30:57.771572 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.771480 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lb8s\" (UniqueName: \"kubernetes.io/projected/719def6b-4f47-4d2d-8665-a7fa3af2311f-kube-api-access-5lb8s\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:57.771572 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:57.771526 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffw8j\" (UniqueName: \"kubernetes.io/projected/56ad34b9-579c-489a-bef6-f595989c2e86-kube-api-access-ffw8j\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:30:58.016417 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:58.016384 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96"] Oct 30 12:30:58.028465 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:58.028388 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-41bdda-6d9fdf4c8c-2jk96"] Oct 30 12:30:58.059910 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:58.059879 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h"] Oct 30 12:30:58.063031 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:58.062999 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-41bdda-69fc77588d-xq77h"] Oct 30 12:30:58.077803 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:58.077767 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:58.187673 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:58.187640 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:30:59.020564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:59.020532 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56ad34b9-579c-489a-bef6-f595989c2e86" path="/var/lib/kubelet/pods/56ad34b9-579c-489a-bef6-f595989c2e86/volumes" Oct 30 12:30:59.020952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:59.020939 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="719def6b-4f47-4d2d-8665-a7fa3af2311f" path="/var/lib/kubelet/pods/719def6b-4f47-4d2d-8665-a7fa3af2311f/volumes" Oct 30 12:30:59.731710 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:30:59.731675 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:01.139435 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:01.139399 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:01.793995 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:01.793961 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:01.909750 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:01.909700 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:02.027468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:02.027439 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:02.449234 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:02.449188 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:03.855212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:03.855162 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:04.739307 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:04.739259 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm"] Oct 30 12:31:04.739564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:04.739534 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" podUID="7e867b36-7f5d-4ad3-9192-0149b302b614" containerName="istio-proxy" containerID="cri-o://59c454eeddc9facc39caadf16a6bf27d896ac964ab87b34a9ae24716f4c0f8e5" gracePeriod=30 Oct 30 12:31:04.996927 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:04.996850 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:31:04.997336 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:04.996920 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" podUID="7e867b36-7f5d-4ad3-9192-0149b302b614" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:31:05.560640 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:05.560590 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:05.814996 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:31:05.814910 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:31:07.373810 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:07.373777 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:09.016593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:09.016554 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:09.133963 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:09.133928 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:09.395535 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:09.395503 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:09.506891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:09.506837 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:10.020027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.019959 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:31:10.039774 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.039749 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-credential-socket\") pod \"7e867b36-7f5d-4ad3-9192-0149b302b614\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " Oct 30 12:31:10.039917 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.039790 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e867b36-7f5d-4ad3-9192-0149b302b614-kube-api-access-twc89\" (UniqueName: \"kubernetes.io/projected/7e867b36-7f5d-4ad3-9192-0149b302b614-kube-api-access-twc89\") pod \"7e867b36-7f5d-4ad3-9192-0149b302b614\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " Oct 30 12:31:10.039917 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.039825 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-workload-certs\") pod \"7e867b36-7f5d-4ad3-9192-0149b302b614\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " Oct 30 12:31:10.039917 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.039889 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-token\" (UniqueName: \"kubernetes.io/projected/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-token\") pod \"7e867b36-7f5d-4ad3-9192-0149b302b614\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " Oct 30 12:31:10.040090 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.039943 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-podinfo\") pod \"7e867b36-7f5d-4ad3-9192-0149b302b614\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " Oct 30 12:31:10.040090 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.039993 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e867b36-7f5d-4ad3-9192-0149b302b614-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e867b36-7f5d-4ad3-9192-0149b302b614-istiod-ca-cert\") pod \"7e867b36-7f5d-4ad3-9192-0149b302b614\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " Oct 30 12:31:10.040090 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.040027 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-credential-socket" pod "7e867b36-7f5d-4ad3-9192-0149b302b614" (UID: "7e867b36-7f5d-4ad3-9192-0149b302b614"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:10.040090 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.040039 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-data\") pod \"7e867b36-7f5d-4ad3-9192-0149b302b614\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " Oct 30 12:31:10.040311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.040118 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-workload-socket\") pod \"7e867b36-7f5d-4ad3-9192-0149b302b614\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " Oct 30 12:31:10.040311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.040222 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-envoy\") pod \"7e867b36-7f5d-4ad3-9192-0149b302b614\" (UID: \"7e867b36-7f5d-4ad3-9192-0149b302b614\") " Oct 30 12:31:10.040311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.040205 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-workload-certs" pod "7e867b36-7f5d-4ad3-9192-0149b302b614" (UID: "7e867b36-7f5d-4ad3-9192-0149b302b614"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:10.040448 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.040334 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-data" pod "7e867b36-7f5d-4ad3-9192-0149b302b614" (UID: "7e867b36-7f5d-4ad3-9192-0149b302b614"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:10.040492 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.040441 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-workload-socket" pod "7e867b36-7f5d-4ad3-9192-0149b302b614" (UID: "7e867b36-7f5d-4ad3-9192-0149b302b614"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:10.040492 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.040472 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e867b36-7f5d-4ad3-9192-0149b302b614-istiod-ca-cert" pod "7e867b36-7f5d-4ad3-9192-0149b302b614" (UID: "7e867b36-7f5d-4ad3-9192-0149b302b614"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:10.041031 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.041010 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:10.041031 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.041029 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:10.041236 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.041039 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:10.041236 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.041051 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e867b36-7f5d-4ad3-9192-0149b302b614-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:10.041236 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.041062 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:10.042410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.042386 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-token" pod "7e867b36-7f5d-4ad3-9192-0149b302b614" (UID: "7e867b36-7f5d-4ad3-9192-0149b302b614"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:10.042499 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.042384 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-podinfo" pod "7e867b36-7f5d-4ad3-9192-0149b302b614" (UID: "7e867b36-7f5d-4ad3-9192-0149b302b614"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:31:10.042869 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.042852 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-envoy" pod "7e867b36-7f5d-4ad3-9192-0149b302b614" (UID: "7e867b36-7f5d-4ad3-9192-0149b302b614"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:10.042910 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.042877 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e867b36-7f5d-4ad3-9192-0149b302b614-kube-api-access-twc89" pod "7e867b36-7f5d-4ad3-9192-0149b302b614" (UID: "7e867b36-7f5d-4ad3-9192-0149b302b614"). InnerVolumeSpecName "kube-api-access-twc89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:10.142101 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.142070 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:10.142101 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.142098 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twc89\" (UniqueName: \"kubernetes.io/projected/7e867b36-7f5d-4ad3-9192-0149b302b614-kube-api-access-twc89\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:10.142314 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.142109 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:10.142314 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.142118 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e867b36-7f5d-4ad3-9192-0149b302b614-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:10.753893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.753862 4413 generic.go:356] "Generic (PLEG): container finished" podID="7e867b36-7f5d-4ad3-9192-0149b302b614" containerID="59c454eeddc9facc39caadf16a6bf27d896ac964ab87b34a9ae24716f4c0f8e5" exitCode=0 Oct 30 12:31:10.754076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.753932 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" Oct 30 12:31:10.754076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.753940 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" event={"ID":"7e867b36-7f5d-4ad3-9192-0149b302b614","Type":"ContainerDied","Data":"59c454eeddc9facc39caadf16a6bf27d896ac964ab87b34a9ae24716f4c0f8e5"} Oct 30 12:31:10.754076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.754006 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm" event={"ID":"7e867b36-7f5d-4ad3-9192-0149b302b614","Type":"ContainerDied","Data":"c32f826f02163c3261b43dc7dc47b867b740d8fe35f7a68c9cd53e85ed10b43c"} Oct 30 12:31:10.754076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.754036 4413 scope.go:122] "RemoveContainer" containerID="59c454eeddc9facc39caadf16a6bf27d896ac964ab87b34a9ae24716f4c0f8e5" Oct 30 12:31:10.765057 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.765023 4413 scope.go:122] "RemoveContainer" containerID="f2f681209ae79ffcbb7e70037a2b2a202e22aa3eb2019c305d28247e0c9e7614" Oct 30 12:31:10.775410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.775393 4413 scope.go:122] "RemoveContainer" containerID="59c454eeddc9facc39caadf16a6bf27d896ac964ab87b34a9ae24716f4c0f8e5" Oct 30 12:31:10.775727 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:31:10.775707 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59c454eeddc9facc39caadf16a6bf27d896ac964ab87b34a9ae24716f4c0f8e5\": container with ID starting with 59c454eeddc9facc39caadf16a6bf27d896ac964ab87b34a9ae24716f4c0f8e5 not found: ID does not exist" containerID="59c454eeddc9facc39caadf16a6bf27d896ac964ab87b34a9ae24716f4c0f8e5" Oct 30 12:31:10.775812 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.775737 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c454eeddc9facc39caadf16a6bf27d896ac964ab87b34a9ae24716f4c0f8e5"} err="failed to get container status \"59c454eeddc9facc39caadf16a6bf27d896ac964ab87b34a9ae24716f4c0f8e5\": rpc error: code = NotFound desc = could not find container \"59c454eeddc9facc39caadf16a6bf27d896ac964ab87b34a9ae24716f4c0f8e5\": container with ID starting with 59c454eeddc9facc39caadf16a6bf27d896ac964ab87b34a9ae24716f4c0f8e5 not found: ID does not exist" Oct 30 12:31:10.775812 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.775757 4413 scope.go:122] "RemoveContainer" containerID="f2f681209ae79ffcbb7e70037a2b2a202e22aa3eb2019c305d28247e0c9e7614" Oct 30 12:31:10.776093 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:31:10.776071 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f681209ae79ffcbb7e70037a2b2a202e22aa3eb2019c305d28247e0c9e7614\": container with ID starting with f2f681209ae79ffcbb7e70037a2b2a202e22aa3eb2019c305d28247e0c9e7614 not found: ID does not exist" containerID="f2f681209ae79ffcbb7e70037a2b2a202e22aa3eb2019c305d28247e0c9e7614" Oct 30 12:31:10.776140 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.776099 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f681209ae79ffcbb7e70037a2b2a202e22aa3eb2019c305d28247e0c9e7614"} err="failed to get container status \"f2f681209ae79ffcbb7e70037a2b2a202e22aa3eb2019c305d28247e0c9e7614\": rpc error: code = NotFound desc = could not find container \"f2f681209ae79ffcbb7e70037a2b2a202e22aa3eb2019c305d28247e0c9e7614\": container with ID starting with f2f681209ae79ffcbb7e70037a2b2a202e22aa3eb2019c305d28247e0c9e7614 not found: ID does not exist" Oct 30 12:31:10.817606 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.817576 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm"] Oct 30 12:31:10.830878 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:10.830845 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-5x7jm"] Oct 30 12:31:11.020198 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:11.020118 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e867b36-7f5d-4ad3-9192-0149b302b614" path="/var/lib/kubelet/pods/7e867b36-7f5d-4ad3-9192-0149b302b614/volumes" Oct 30 12:31:11.838112 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:11.838084 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:13.375115 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.375084 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72"] Oct 30 12:31:13.375597 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.375349 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" podUID="6159985a-714a-4a43-8645-9865f379b1a3" containerName="maas-api" containerID="cri-o://a9e7dc30e7451abc6275a5748b6b8a0eb7a79bca69666e262879d19f978584a6" gracePeriod=30 Oct 30 12:31:13.517579 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.517544 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:13.653276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.653170 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:31:13.656053 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.656025 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8"] Oct 30 12:31:13.656222 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.656204 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" podUID="6662310d-7d7f-424e-a1c8-45988ed0db96" containerName="istio-proxy" containerID="cri-o://875f3ceae94422cf23a6b22b7f20badedf7e530818f607535c7c2f908f403c35" gracePeriod=30 Oct 30 12:31:13.686387 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.686353 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6159985a-714a-4a43-8645-9865f379b1a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6159985a-714a-4a43-8645-9865f379b1a3-metrics-certs\") pod \"6159985a-714a-4a43-8645-9865f379b1a3\" (UID: \"6159985a-714a-4a43-8645-9865f379b1a3\") " Oct 30 12:31:13.686585 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.686455 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6159985a-714a-4a43-8645-9865f379b1a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6159985a-714a-4a43-8645-9865f379b1a3-openshift-service-ca\") pod \"6159985a-714a-4a43-8645-9865f379b1a3\" (UID: \"6159985a-714a-4a43-8645-9865f379b1a3\") " Oct 30 12:31:13.686585 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.686569 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6159985a-714a-4a43-8645-9865f379b1a3-kube-api-access-qb7vp\" (UniqueName: \"kubernetes.io/projected/6159985a-714a-4a43-8645-9865f379b1a3-kube-api-access-qb7vp\") pod \"6159985a-714a-4a43-8645-9865f379b1a3\" (UID: \"6159985a-714a-4a43-8645-9865f379b1a3\") " Oct 30 12:31:13.686704 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.686667 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6159985a-714a-4a43-8645-9865f379b1a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6159985a-714a-4a43-8645-9865f379b1a3-maas-api-tls\") pod \"6159985a-714a-4a43-8645-9865f379b1a3\" (UID: \"6159985a-714a-4a43-8645-9865f379b1a3\") " Oct 30 12:31:13.686920 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.686890 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6159985a-714a-4a43-8645-9865f379b1a3-openshift-service-ca" pod "6159985a-714a-4a43-8645-9865f379b1a3" (UID: "6159985a-714a-4a43-8645-9865f379b1a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:13.687789 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.687765 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6159985a-714a-4a43-8645-9865f379b1a3-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:13.690209 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.690181 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6159985a-714a-4a43-8645-9865f379b1a3-kube-api-access-qb7vp" pod "6159985a-714a-4a43-8645-9865f379b1a3" (UID: "6159985a-714a-4a43-8645-9865f379b1a3"). InnerVolumeSpecName "kube-api-access-qb7vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:13.690410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.690380 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6159985a-714a-4a43-8645-9865f379b1a3-maas-api-tls" pod "6159985a-714a-4a43-8645-9865f379b1a3" (UID: "6159985a-714a-4a43-8645-9865f379b1a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:13.691081 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.691000 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6159985a-714a-4a43-8645-9865f379b1a3-metrics-certs" pod "6159985a-714a-4a43-8645-9865f379b1a3" (UID: "6159985a-714a-4a43-8645-9865f379b1a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:13.774690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.774649 4413 generic.go:356] "Generic (PLEG): container finished" podID="6159985a-714a-4a43-8645-9865f379b1a3" containerID="a9e7dc30e7451abc6275a5748b6b8a0eb7a79bca69666e262879d19f978584a6" exitCode=0 Oct 30 12:31:13.774888 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.774703 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" Oct 30 12:31:13.774888 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.774736 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" event={"ID":"6159985a-714a-4a43-8645-9865f379b1a3","Type":"ContainerDied","Data":"a9e7dc30e7451abc6275a5748b6b8a0eb7a79bca69666e262879d19f978584a6"} Oct 30 12:31:13.774888 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.774805 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72" event={"ID":"6159985a-714a-4a43-8645-9865f379b1a3","Type":"ContainerDied","Data":"8e96385275172fb6bb49999423f541cf97c9bfd30095203e4900fd79acbc4b7c"} Oct 30 12:31:13.774888 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.774834 4413 scope.go:122] "RemoveContainer" containerID="a9e7dc30e7451abc6275a5748b6b8a0eb7a79bca69666e262879d19f978584a6" Oct 30 12:31:13.785616 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.785597 4413 scope.go:122] "RemoveContainer" containerID="a9e7dc30e7451abc6275a5748b6b8a0eb7a79bca69666e262879d19f978584a6" Oct 30 12:31:13.785990 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:31:13.785965 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9e7dc30e7451abc6275a5748b6b8a0eb7a79bca69666e262879d19f978584a6\": container with ID starting with a9e7dc30e7451abc6275a5748b6b8a0eb7a79bca69666e262879d19f978584a6 not found: ID does not exist" containerID="a9e7dc30e7451abc6275a5748b6b8a0eb7a79bca69666e262879d19f978584a6" Oct 30 12:31:13.786063 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.786003 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e7dc30e7451abc6275a5748b6b8a0eb7a79bca69666e262879d19f978584a6"} err="failed to get container status \"a9e7dc30e7451abc6275a5748b6b8a0eb7a79bca69666e262879d19f978584a6\": rpc error: code = NotFound desc = could not find container \"a9e7dc30e7451abc6275a5748b6b8a0eb7a79bca69666e262879d19f978584a6\": container with ID starting with a9e7dc30e7451abc6275a5748b6b8a0eb7a79bca69666e262879d19f978584a6 not found: ID does not exist" Oct 30 12:31:13.788899 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.788880 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6159985a-714a-4a43-8645-9865f379b1a3-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:13.788899 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.788900 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb7vp\" (UniqueName: \"kubernetes.io/projected/6159985a-714a-4a43-8645-9865f379b1a3-kube-api-access-qb7vp\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:13.789037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.788910 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6159985a-714a-4a43-8645-9865f379b1a3-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:13.825011 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.824976 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72"] Oct 30 12:31:13.828602 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.828575 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-41bdda-6dd6f8c459-5lm72"] Oct 30 12:31:13.966651 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:13.966565 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:14.114405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:14.114361 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:14.310099 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:14.310017 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:14.672323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:14.672277 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:14.911765 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:14.911735 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:15.021624 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:15.021545 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6159985a-714a-4a43-8645-9865f379b1a3" path="/var/lib/kubelet/pods/6159985a-714a-4a43-8645-9865f379b1a3/volumes" Oct 30 12:31:15.484705 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:15.484678 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:15.604674 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:15.604643 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:16.690091 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:16.690058 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:17.503542 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.503504 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:31:17.503702 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.503583 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" podUID="6662310d-7d7f-424e-a1c8-45988ed0db96" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:31:17.534885 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.534854 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h"] Oct 30 12:31:17.535528 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535510 4413 cpu_manager.go:414] "containerMap: removing container" podUID="7e867b36-7f5d-4ad3-9192-0149b302b614" containerName="istio-proxy" Oct 30 12:31:17.535528 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535529 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e867b36-7f5d-4ad3-9192-0149b302b614" containerName="istio-proxy" Oct 30 12:31:17.535723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535544 4413 cpu_manager.go:414] "containerMap: removing container" podUID="56ad34b9-579c-489a-bef6-f595989c2e86" containerName="payload-processing" Oct 30 12:31:17.535723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535554 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56ad34b9-579c-489a-bef6-f595989c2e86" containerName="payload-processing" Oct 30 12:31:17.535723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535567 4413 cpu_manager.go:414] "containerMap: removing container" podUID="d164b031-7ba1-48d2-9890-01e3e941ad3e" containerName="cleanup" Oct 30 12:31:17.535723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535574 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d164b031-7ba1-48d2-9890-01e3e941ad3e" containerName="cleanup" Oct 30 12:31:17.535723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535597 4413 cpu_manager.go:414] "containerMap: removing container" podUID="d164b031-7ba1-48d2-9890-01e3e941ad3e" containerName="cleanup" Oct 30 12:31:17.535723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535604 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d164b031-7ba1-48d2-9890-01e3e941ad3e" containerName="cleanup" Oct 30 12:31:17.535723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535629 4413 cpu_manager.go:414] "containerMap: removing container" podUID="d8af5dd5-cd57-4892-86dd-54f2b3b203a0" containerName="payload-pre-processing" Oct 30 12:31:17.535723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535637 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8af5dd5-cd57-4892-86dd-54f2b3b203a0" containerName="payload-pre-processing" Oct 30 12:31:17.535723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535651 4413 cpu_manager.go:414] "containerMap: removing container" podUID="2d20b6be-6972-426c-afff-bb5ae5f93a6e" containerName="payload-processing" Oct 30 12:31:17.535723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535657 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d20b6be-6972-426c-afff-bb5ae5f93a6e" containerName="payload-processing" Oct 30 12:31:17.535723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535679 4413 cpu_manager.go:414] "containerMap: removing container" podUID="6159985a-714a-4a43-8645-9865f379b1a3" containerName="maas-api" Oct 30 12:31:17.535723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535686 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6159985a-714a-4a43-8645-9865f379b1a3" containerName="maas-api" Oct 30 12:31:17.535723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535703 4413 cpu_manager.go:414] "containerMap: removing container" podUID="799fe64f-b3d4-43bf-b7ae-04833737b666" containerName="discovery" Oct 30 12:31:17.535723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535710 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="799fe64f-b3d4-43bf-b7ae-04833737b666" containerName="discovery" Oct 30 12:31:17.536486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535734 4413 cpu_manager.go:414] "containerMap: removing container" podUID="719def6b-4f47-4d2d-8665-a7fa3af2311f" containerName="payload-pre-processing" Oct 30 12:31:17.536486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535742 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="719def6b-4f47-4d2d-8665-a7fa3af2311f" containerName="payload-pre-processing" Oct 30 12:31:17.536486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535760 4413 cpu_manager.go:414] "containerMap: removing container" podUID="a1a91278-7714-4ff4-a35a-5f573a09497a" containerName="maas-api" Oct 30 12:31:17.536486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535767 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a91278-7714-4ff4-a35a-5f573a09497a" containerName="maas-api" Oct 30 12:31:17.536486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535927 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="d164b031-7ba1-48d2-9890-01e3e941ad3e" containerName="cleanup" Oct 30 12:31:17.536486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535941 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e867b36-7f5d-4ad3-9192-0149b302b614" containerName="istio-proxy" Oct 30 12:31:17.536486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535956 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8af5dd5-cd57-4892-86dd-54f2b3b203a0" containerName="payload-pre-processing" Oct 30 12:31:17.536486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535967 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d20b6be-6972-426c-afff-bb5ae5f93a6e" containerName="payload-processing" Oct 30 12:31:17.536486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535979 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="6159985a-714a-4a43-8645-9865f379b1a3" containerName="maas-api" Oct 30 12:31:17.536486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.535988 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="799fe64f-b3d4-43bf-b7ae-04833737b666" containerName="discovery" Oct 30 12:31:17.536486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.536000 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="719def6b-4f47-4d2d-8665-a7fa3af2311f" containerName="payload-pre-processing" Oct 30 12:31:17.536486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.536011 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a91278-7714-4ff4-a35a-5f573a09497a" containerName="maas-api" Oct 30 12:31:17.536486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.536020 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="d164b031-7ba1-48d2-9890-01e3e941ad3e" containerName="cleanup" Oct 30 12:31:17.536486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.536030 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="56ad34b9-579c-489a-bef6-f595989c2e86" containerName="payload-processing" Oct 30 12:31:17.536486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.536040 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="d164b031-7ba1-48d2-9890-01e3e941ad3e" containerName="cleanup" Oct 30 12:31:17.539884 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.539853 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.558676 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.558648 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h"] Oct 30 12:31:17.569189 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.569165 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-c44eee\"/\"openshift-service-ca.crt\"" Oct 30 12:31:17.579657 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.579633 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-c44eee\"/\"default-dockercfg-2bmnm\"" Oct 30 12:31:17.579657 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.579640 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-c44eee\"/\"kube-root-ca.crt\"" Oct 30 12:31:17.579834 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.579723 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-c44eee\"/\"praxis-route-c44eee-kserve-self-signed-certs\"" Oct 30 12:31:17.634671 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.634631 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:17.645106 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.645073 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85ck6\" (UniqueName: \"kubernetes.io/projected/969651c2-7e9e-4b32-8e86-be72e29bd06b-kube-api-access-85ck6\") pod \"praxis-route-c44eee-kserve-566795949c-hz74h\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.645305 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.645114 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-dshm\") pod \"praxis-route-c44eee-kserve-566795949c-hz74h\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.645305 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.645193 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-home\") pod \"praxis-route-c44eee-kserve-566795949c-hz74h\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.645305 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.645267 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-tmp-dir\") pod \"praxis-route-c44eee-kserve-566795949c-hz74h\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.645305 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.645305 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-model-cache\") pod \"praxis-route-c44eee-kserve-566795949c-hz74h\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.645469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.645449 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969651c2-7e9e-4b32-8e86-be72e29bd06b-tls-certs\") pod \"praxis-route-c44eee-kserve-566795949c-hz74h\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.747177 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.747145 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85ck6\" (UniqueName: \"kubernetes.io/projected/969651c2-7e9e-4b32-8e86-be72e29bd06b-kube-api-access-85ck6\") pod \"praxis-route-c44eee-kserve-566795949c-hz74h\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.747177 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.747188 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-dshm\") pod \"praxis-route-c44eee-kserve-566795949c-hz74h\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.747749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.747342 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-home\") pod \"praxis-route-c44eee-kserve-566795949c-hz74h\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.747749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.747422 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-tmp-dir\") pod \"praxis-route-c44eee-kserve-566795949c-hz74h\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.747749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.747479 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-model-cache\") pod \"praxis-route-c44eee-kserve-566795949c-hz74h\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.747749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.747635 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969651c2-7e9e-4b32-8e86-be72e29bd06b-tls-certs\") pod \"praxis-route-c44eee-kserve-566795949c-hz74h\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.747749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.747691 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-home\") pod \"praxis-route-c44eee-kserve-566795949c-hz74h\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.748040 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.747748 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-tmp-dir\") pod \"praxis-route-c44eee-kserve-566795949c-hz74h\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.748040 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.747759 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-model-cache\") pod \"praxis-route-c44eee-kserve-566795949c-hz74h\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.749727 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.749706 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-dshm\") pod \"praxis-route-c44eee-kserve-566795949c-hz74h\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.750358 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.750337 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969651c2-7e9e-4b32-8e86-be72e29bd06b-tls-certs\") pod \"praxis-route-c44eee-kserve-566795949c-hz74h\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.769642 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.769584 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85ck6\" (UniqueName: \"kubernetes.io/projected/969651c2-7e9e-4b32-8e86-be72e29bd06b-kube-api-access-85ck6\") pod \"praxis-route-c44eee-kserve-566795949c-hz74h\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:17.851649 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:17.851609 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:18.158282 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:18.158238 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h"] Oct 30 12:31:18.159237 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:31:18.159211 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod969651c2_7e9e_4b32_8e86_be72e29bd06b.slice/crio-de5bc641a7c2e2db055082139bcbfce1e5287d96ee80c2f16134baa655f7045d WatchSource:0}: Error finding container de5bc641a7c2e2db055082139bcbfce1e5287d96ee80c2f16134baa655f7045d: Status 404 returned error can't find the container with id de5bc641a7c2e2db055082139bcbfce1e5287d96ee80c2f16134baa655f7045d Oct 30 12:31:18.645433 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:18.645401 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:18.816613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:18.816570 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" event={"ID":"969651c2-7e9e-4b32-8e86-be72e29bd06b","Type":"ContainerStarted","Data":"465bf9d43552f6e6a55098c0d1f304089e33fc3bee449dcdaeed3bde13e5c767"} Oct 30 12:31:18.816613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:18.816620 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" event={"ID":"969651c2-7e9e-4b32-8e86-be72e29bd06b","Type":"ContainerStarted","Data":"de5bc641a7c2e2db055082139bcbfce1e5287d96ee80c2f16134baa655f7045d"} Oct 30 12:31:18.818447 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:18.818421 4413 generic.go:356] "Generic (PLEG): container finished" podID="6662310d-7d7f-424e-a1c8-45988ed0db96" containerID="875f3ceae94422cf23a6b22b7f20badedf7e530818f607535c7c2f908f403c35" exitCode=0 Oct 30 12:31:18.818571 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:18.818461 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" event={"ID":"6662310d-7d7f-424e-a1c8-45988ed0db96","Type":"ContainerDied","Data":"875f3ceae94422cf23a6b22b7f20badedf7e530818f607535c7c2f908f403c35"} Oct 30 12:31:18.852693 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:18.852669 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:18.861199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:18.861148 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" podStartSLOduration=1.861133824 podStartE2EDuration="1.861133824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:18.858364465 +0000 UTC m=+2231.968207347" watchObservedRunningTime="2026-10-30 12:31:18.861133824 +0000 UTC m=+2231.970976705" Oct 30 12:31:18.930219 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:18.930199 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:31:19.063431 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.063402 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-envoy\") pod \"6662310d-7d7f-424e-a1c8-45988ed0db96\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " Oct 30 12:31:19.063587 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.063523 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-podinfo\") pod \"6662310d-7d7f-424e-a1c8-45988ed0db96\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " Oct 30 12:31:19.063640 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.063594 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6662310d-7d7f-424e-a1c8-45988ed0db96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6662310d-7d7f-424e-a1c8-45988ed0db96-istiod-ca-cert\") pod \"6662310d-7d7f-424e-a1c8-45988ed0db96\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " Oct 30 12:31:19.063640 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.063618 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-data\") pod \"6662310d-7d7f-424e-a1c8-45988ed0db96\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " Oct 30 12:31:19.063731 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.063643 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-token\" (UniqueName: \"kubernetes.io/projected/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-token\") pod \"6662310d-7d7f-424e-a1c8-45988ed0db96\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " Oct 30 12:31:19.063731 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.063707 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6662310d-7d7f-424e-a1c8-45988ed0db96-kube-api-access-xbpvh\" (UniqueName: \"kubernetes.io/projected/6662310d-7d7f-424e-a1c8-45988ed0db96-kube-api-access-xbpvh\") pod \"6662310d-7d7f-424e-a1c8-45988ed0db96\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " Oct 30 12:31:19.063844 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.063751 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-credential-socket\") pod \"6662310d-7d7f-424e-a1c8-45988ed0db96\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " Oct 30 12:31:19.063844 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.063800 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-workload-certs\") pod \"6662310d-7d7f-424e-a1c8-45988ed0db96\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " Oct 30 12:31:19.063844 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.063834 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-workload-socket\") pod \"6662310d-7d7f-424e-a1c8-45988ed0db96\" (UID: \"6662310d-7d7f-424e-a1c8-45988ed0db96\") " Oct 30 12:31:19.063997 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.063941 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-data" pod "6662310d-7d7f-424e-a1c8-45988ed0db96" (UID: "6662310d-7d7f-424e-a1c8-45988ed0db96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:19.063997 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.063969 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6662310d-7d7f-424e-a1c8-45988ed0db96-istiod-ca-cert" pod "6662310d-7d7f-424e-a1c8-45988ed0db96" (UID: "6662310d-7d7f-424e-a1c8-45988ed0db96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:19.064239 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.064162 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-credential-socket" pod "6662310d-7d7f-424e-a1c8-45988ed0db96" (UID: "6662310d-7d7f-424e-a1c8-45988ed0db96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:19.064513 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.064486 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-workload-certs" pod "6662310d-7d7f-424e-a1c8-45988ed0db96" (UID: "6662310d-7d7f-424e-a1c8-45988ed0db96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:19.064593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.064495 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6662310d-7d7f-424e-a1c8-45988ed0db96-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:19.064593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.064547 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:19.064593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.064566 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:19.064705 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.064639 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-workload-socket" pod "6662310d-7d7f-424e-a1c8-45988ed0db96" (UID: "6662310d-7d7f-424e-a1c8-45988ed0db96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:19.065793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.065769 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-envoy" pod "6662310d-7d7f-424e-a1c8-45988ed0db96" (UID: "6662310d-7d7f-424e-a1c8-45988ed0db96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:19.066163 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.066146 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6662310d-7d7f-424e-a1c8-45988ed0db96-kube-api-access-xbpvh" pod "6662310d-7d7f-424e-a1c8-45988ed0db96" (UID: "6662310d-7d7f-424e-a1c8-45988ed0db96"). InnerVolumeSpecName "kube-api-access-xbpvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:19.066385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.066365 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-podinfo" pod "6662310d-7d7f-424e-a1c8-45988ed0db96" (UID: "6662310d-7d7f-424e-a1c8-45988ed0db96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:31:19.066473 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.066379 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-token" pod "6662310d-7d7f-424e-a1c8-45988ed0db96" (UID: "6662310d-7d7f-424e-a1c8-45988ed0db96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:19.165854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.165772 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:19.165854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.165801 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:19.165854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.165814 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbpvh\" (UniqueName: \"kubernetes.io/projected/6662310d-7d7f-424e-a1c8-45988ed0db96-kube-api-access-xbpvh\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:19.165854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.165828 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:19.165854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.165837 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:19.165854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.165844 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6662310d-7d7f-424e-a1c8-45988ed0db96-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:19.620354 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.620242 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:19.829395 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.829365 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" Oct 30 12:31:19.829395 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.829375 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8" event={"ID":"6662310d-7d7f-424e-a1c8-45988ed0db96","Type":"ContainerDied","Data":"968e98f4cf33a5a736fdca643586162e8373b696bdbcb6ec9378bd843bb6df0b"} Oct 30 12:31:19.829975 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.829489 4413 scope.go:122] "RemoveContainer" containerID="875f3ceae94422cf23a6b22b7f20badedf7e530818f607535c7c2f908f403c35" Oct 30 12:31:19.881816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.881789 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8"] Oct 30 12:31:19.893731 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:19.893705 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w9qb8"] Oct 30 12:31:20.348933 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:20.348857 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:21.020144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:21.020114 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6662310d-7d7f-424e-a1c8-45988ed0db96" path="/var/lib/kubelet/pods/6662310d-7d7f-424e-a1c8-45988ed0db96/volumes" Oct 30 12:31:21.080396 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:21.080363 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:21.575651 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:21.575616 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:21.701901 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:21.701841 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:21.937981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:21.937950 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:22.054852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:22.054819 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:23.799934 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:23.799902 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:25.457811 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:25.457778 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:27.160417 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:27.160378 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:27.277045 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:27.277010 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:27.851947 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:27.851921 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:27.864739 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:27.864716 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:27.893840 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:27.893813 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:31:28.862198 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:28.862162 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:30.240971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.240935 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5"] Oct 30 12:31:30.241600 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.241210 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" podUID="51fa9579-e6a6-4c1d-a867-da1bd42fa91b" containerName="maas-api" containerID="cri-o://f336216edbc704d40526487ecf7300f539b95d72ff2d32582a0b16ee6100746f" gracePeriod=30 Oct 30 12:31:30.316115 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.316085 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:30.497956 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.497864 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:30.540692 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.540666 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" Oct 30 12:31:30.622066 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.622031 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-openshift-service-ca\") pod \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\" (UID: \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\") " Oct 30 12:31:30.622286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.622172 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-maas-api-tls\") pod \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\" (UID: \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\") " Oct 30 12:31:30.622286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.622226 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-kube-api-access-gkx8s\" (UniqueName: \"kubernetes.io/projected/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-kube-api-access-gkx8s\") pod \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\" (UID: \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\") " Oct 30 12:31:30.622286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.622275 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-metrics-certs\") pod \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\" (UID: \"51fa9579-e6a6-4c1d-a867-da1bd42fa91b\") " Oct 30 12:31:30.622520 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.622497 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-openshift-service-ca" pod "51fa9579-e6a6-4c1d-a867-da1bd42fa91b" (UID: "51fa9579-e6a6-4c1d-a867-da1bd42fa91b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:30.622983 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.622884 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:30.624582 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.624549 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-maas-api-tls" pod "51fa9579-e6a6-4c1d-a867-da1bd42fa91b" (UID: "51fa9579-e6a6-4c1d-a867-da1bd42fa91b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:30.624690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.624655 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-kube-api-access-gkx8s" pod "51fa9579-e6a6-4c1d-a867-da1bd42fa91b" (UID: "51fa9579-e6a6-4c1d-a867-da1bd42fa91b"). InnerVolumeSpecName "kube-api-access-gkx8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:30.624948 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.624928 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-metrics-certs" pod "51fa9579-e6a6-4c1d-a867-da1bd42fa91b" (UID: "51fa9579-e6a6-4c1d-a867-da1bd42fa91b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:30.641296 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.641269 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:30.724781 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.724744 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:30.724781 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.724783 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkx8s\" (UniqueName: \"kubernetes.io/projected/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-kube-api-access-gkx8s\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:30.725057 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.724798 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51fa9579-e6a6-4c1d-a867-da1bd42fa91b-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:30.919297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.919233 4413 generic.go:356] "Generic (PLEG): container finished" podID="51fa9579-e6a6-4c1d-a867-da1bd42fa91b" containerID="f336216edbc704d40526487ecf7300f539b95d72ff2d32582a0b16ee6100746f" exitCode=0 Oct 30 12:31:30.919297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.919280 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" event={"ID":"51fa9579-e6a6-4c1d-a867-da1bd42fa91b","Type":"ContainerDied","Data":"f336216edbc704d40526487ecf7300f539b95d72ff2d32582a0b16ee6100746f"} Oct 30 12:31:30.919297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.919297 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" Oct 30 12:31:30.919569 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.919334 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5" event={"ID":"51fa9579-e6a6-4c1d-a867-da1bd42fa91b","Type":"ContainerDied","Data":"e795a312553fef2623e2448e974cf692402e5423b4639fb11559d7cc7ed50a15"} Oct 30 12:31:30.919569 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.919358 4413 scope.go:122] "RemoveContainer" containerID="f336216edbc704d40526487ecf7300f539b95d72ff2d32582a0b16ee6100746f" Oct 30 12:31:30.930156 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.930132 4413 scope.go:122] "RemoveContainer" containerID="f336216edbc704d40526487ecf7300f539b95d72ff2d32582a0b16ee6100746f" Oct 30 12:31:30.930474 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:31:30.930452 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f336216edbc704d40526487ecf7300f539b95d72ff2d32582a0b16ee6100746f\": container with ID starting with f336216edbc704d40526487ecf7300f539b95d72ff2d32582a0b16ee6100746f not found: ID does not exist" containerID="f336216edbc704d40526487ecf7300f539b95d72ff2d32582a0b16ee6100746f" Oct 30 12:31:30.930558 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:30.930485 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f336216edbc704d40526487ecf7300f539b95d72ff2d32582a0b16ee6100746f"} err="failed to get container status \"f336216edbc704d40526487ecf7300f539b95d72ff2d32582a0b16ee6100746f\": rpc error: code = NotFound desc = could not find container \"f336216edbc704d40526487ecf7300f539b95d72ff2d32582a0b16ee6100746f\": container with ID starting with f336216edbc704d40526487ecf7300f539b95d72ff2d32582a0b16ee6100746f not found: ID does not exist" Oct 30 12:31:31.023108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:31.023078 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5"] Oct 30 12:31:31.026369 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:31.026343 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f78d4-d4f74d764-xfbv5"] Oct 30 12:31:31.066510 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:31.066482 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:31.230182 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:31.230078 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:31.640201 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:31.640169 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:32.984110 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:32.984081 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4"] Oct 30 12:31:33.027757 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:33.027722 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51fa9579-e6a6-4c1d-a867-da1bd42fa91b" path="/var/lib/kubelet/pods/51fa9579-e6a6-4c1d-a867-da1bd42fa91b/volumes" Oct 30 12:31:33.069579 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:33.069531 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:33.196289 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:33.196241 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:35.150659 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:35.150623 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:35.814389 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:31:35.814356 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:31:36.739552 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:36.739520 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:37.984849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:37.984788 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" podUID="78cbff45-e07e-4847-af31-667665133aeb" containerName="payload-processing" containerID="cri-o://3156188d4887b5123f85ddc62e050f10e61bee989e032c80a01ead4ea540d854" gracePeriod=25 Oct 30 12:31:37.996739 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:37.996706 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:38.042688 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:31:38.042525 4413 logging.go:55] [core] [Channel #905 SubChannel #906]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.120:50052", ServerName: "10.133.0.120:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.120:50052: connect: connection refused" Oct 30 12:31:38.116272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.116155 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:38.272655 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.272626 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" Oct 30 12:31:38.315911 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.315878 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78cbff45-e07e-4847-af31-667665133aeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78cbff45-e07e-4847-af31-667665133aeb-plugins-config-volume\") pod \"78cbff45-e07e-4847-af31-667665133aeb\" (UID: \"78cbff45-e07e-4847-af31-667665133aeb\") " Oct 30 12:31:38.316231 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.316209 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78cbff45-e07e-4847-af31-667665133aeb-kube-api-access-654rt\" (UniqueName: \"kubernetes.io/projected/78cbff45-e07e-4847-af31-667665133aeb-kube-api-access-654rt\") pod \"78cbff45-e07e-4847-af31-667665133aeb\" (UID: \"78cbff45-e07e-4847-af31-667665133aeb\") " Oct 30 12:31:38.316465 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.316329 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78cbff45-e07e-4847-af31-667665133aeb-plugins-config-volume" pod "78cbff45-e07e-4847-af31-667665133aeb" (UID: "78cbff45-e07e-4847-af31-667665133aeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:38.317164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.317142 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78cbff45-e07e-4847-af31-667665133aeb-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:38.338156 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.338098 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78cbff45-e07e-4847-af31-667665133aeb-kube-api-access-654rt" pod "78cbff45-e07e-4847-af31-667665133aeb" (UID: "78cbff45-e07e-4847-af31-667665133aeb"). InnerVolumeSpecName "kube-api-access-654rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:38.367330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.367298 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj"] Oct 30 12:31:38.367793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.367760 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" podUID="0250c9a9-b527-47e2-8d57-f221fcbd8e19" containerName="istio-proxy" containerID="cri-o://7d4e0571053faf2806ac90492208c9dbc0e2f8ae4e384ac966355c52be042886" gracePeriod=30 Oct 30 12:31:38.400527 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.400502 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:38.418491 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.418449 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-654rt\" (UniqueName: \"kubernetes.io/projected/78cbff45-e07e-4847-af31-667665133aeb-kube-api-access-654rt\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:38.759013 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.758971 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:38.972051 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.972019 4413 generic.go:356] "Generic (PLEG): container finished" podID="78cbff45-e07e-4847-af31-667665133aeb" containerID="3156188d4887b5123f85ddc62e050f10e61bee989e032c80a01ead4ea540d854" exitCode=0 Oct 30 12:31:38.972230 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.972067 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" Oct 30 12:31:38.972230 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.972098 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" event={"ID":"78cbff45-e07e-4847-af31-667665133aeb","Type":"ContainerDied","Data":"3156188d4887b5123f85ddc62e050f10e61bee989e032c80a01ead4ea540d854"} Oct 30 12:31:38.972230 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.972148 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" event={"ID":"78cbff45-e07e-4847-af31-667665133aeb","Type":"ContainerDied","Data":"0416d69315863e5ab8e7800094e963bdc192d429021eaf3fb8c1cb07f215224e"} Oct 30 12:31:38.972230 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.972174 4413 scope.go:122] "RemoveContainer" containerID="3156188d4887b5123f85ddc62e050f10e61bee989e032c80a01ead4ea540d854" Oct 30 12:31:38.982347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.982332 4413 scope.go:122] "RemoveContainer" containerID="3156188d4887b5123f85ddc62e050f10e61bee989e032c80a01ead4ea540d854" Oct 30 12:31:38.982640 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:31:38.982623 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3156188d4887b5123f85ddc62e050f10e61bee989e032c80a01ead4ea540d854\": container with ID starting with 3156188d4887b5123f85ddc62e050f10e61bee989e032c80a01ead4ea540d854 not found: ID does not exist" containerID="3156188d4887b5123f85ddc62e050f10e61bee989e032c80a01ead4ea540d854" Oct 30 12:31:38.982721 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:38.982647 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3156188d4887b5123f85ddc62e050f10e61bee989e032c80a01ead4ea540d854"} err="failed to get container status \"3156188d4887b5123f85ddc62e050f10e61bee989e032c80a01ead4ea540d854\": rpc error: code = NotFound desc = could not find container \"3156188d4887b5123f85ddc62e050f10e61bee989e032c80a01ead4ea540d854\": container with ID starting with 3156188d4887b5123f85ddc62e050f10e61bee989e032c80a01ead4ea540d854 not found: ID does not exist" Oct 30 12:31:39.036352 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:39.031744 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4"] Oct 30 12:31:39.042370 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:39.042346 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4"] Oct 30 12:31:39.042659 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:39.042632 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e9675-b4db7fc48-gcbd4" podUID="78cbff45-e07e-4847-af31-667665133aeb" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.120:50052\" within 1s: context deadline exceeded" Oct 30 12:31:40.025978 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.025947 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:40.181740 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.181701 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr"] Oct 30 12:31:40.182238 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.181961 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" podUID="5b83871e-854a-4e91-8857-6c63753834db" containerName="maas-api" containerID="cri-o://265a089eb19dcf450bd61fa1a8a432ac18198d3d49cc8dded812b0aebecc1a92" gracePeriod=30 Oct 30 12:31:40.243324 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.243294 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:40.434024 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.433993 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" Oct 30 12:31:40.445116 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.445077 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84"] Oct 30 12:31:40.445705 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.445678 4413 cpu_manager.go:414] "containerMap: removing container" podUID="7e867b36-7f5d-4ad3-9192-0149b302b614" containerName="istio-proxy" Oct 30 12:31:40.445822 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.445707 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e867b36-7f5d-4ad3-9192-0149b302b614" containerName="istio-proxy" Oct 30 12:31:40.445822 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.445727 4413 cpu_manager.go:414] "containerMap: removing container" podUID="5b83871e-854a-4e91-8857-6c63753834db" containerName="maas-api" Oct 30 12:31:40.445822 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.445734 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b83871e-854a-4e91-8857-6c63753834db" containerName="maas-api" Oct 30 12:31:40.445822 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.445764 4413 cpu_manager.go:414] "containerMap: removing container" podUID="d164b031-7ba1-48d2-9890-01e3e941ad3e" containerName="cleanup" Oct 30 12:31:40.445822 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.445773 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d164b031-7ba1-48d2-9890-01e3e941ad3e" containerName="cleanup" Oct 30 12:31:40.445822 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.445802 4413 cpu_manager.go:414] "containerMap: removing container" podUID="51fa9579-e6a6-4c1d-a867-da1bd42fa91b" containerName="maas-api" Oct 30 12:31:40.445822 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.445809 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51fa9579-e6a6-4c1d-a867-da1bd42fa91b" containerName="maas-api" Oct 30 12:31:40.446196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.445830 4413 cpu_manager.go:414] "containerMap: removing container" podUID="78cbff45-e07e-4847-af31-667665133aeb" containerName="payload-processing" Oct 30 12:31:40.446196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.445837 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78cbff45-e07e-4847-af31-667665133aeb" containerName="payload-processing" Oct 30 12:31:40.446196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.445853 4413 cpu_manager.go:414] "containerMap: removing container" podUID="6662310d-7d7f-424e-a1c8-45988ed0db96" containerName="istio-proxy" Oct 30 12:31:40.446196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.445860 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6662310d-7d7f-424e-a1c8-45988ed0db96" containerName="istio-proxy" Oct 30 12:31:40.446196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.446027 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b83871e-854a-4e91-8857-6c63753834db" containerName="maas-api" Oct 30 12:31:40.446196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.446045 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e867b36-7f5d-4ad3-9192-0149b302b614" containerName="istio-proxy" Oct 30 12:31:40.446196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.446057 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="51fa9579-e6a6-4c1d-a867-da1bd42fa91b" containerName="maas-api" Oct 30 12:31:40.446196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.446067 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="78cbff45-e07e-4847-af31-667665133aeb" containerName="payload-processing" Oct 30 12:31:40.446196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.446076 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="6662310d-7d7f-424e-a1c8-45988ed0db96" containerName="istio-proxy" Oct 30 12:31:40.448502 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.448476 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.467480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.467454 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b83871e-854a-4e91-8857-6c63753834db-kube-api-access-zd9lq\" (UniqueName: \"kubernetes.io/projected/5b83871e-854a-4e91-8857-6c63753834db-kube-api-access-zd9lq\") pod \"5b83871e-854a-4e91-8857-6c63753834db\" (UID: \"5b83871e-854a-4e91-8857-6c63753834db\") " Oct 30 12:31:40.467662 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.467642 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b83871e-854a-4e91-8857-6c63753834db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b83871e-854a-4e91-8857-6c63753834db-openshift-service-ca\") pod \"5b83871e-854a-4e91-8857-6c63753834db\" (UID: \"5b83871e-854a-4e91-8857-6c63753834db\") " Oct 30 12:31:40.468064 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.467682 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b83871e-854a-4e91-8857-6c63753834db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b83871e-854a-4e91-8857-6c63753834db-metrics-certs\") pod \"5b83871e-854a-4e91-8857-6c63753834db\" (UID: \"5b83871e-854a-4e91-8857-6c63753834db\") " Oct 30 12:31:40.468064 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.467716 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b83871e-854a-4e91-8857-6c63753834db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b83871e-854a-4e91-8857-6c63753834db-maas-api-tls\") pod \"5b83871e-854a-4e91-8857-6c63753834db\" (UID: \"5b83871e-854a-4e91-8857-6c63753834db\") " Oct 30 12:31:40.468227 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.468085 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-0172c6-openshift-default-dockercfg-c88d4\"" Oct 30 12:31:40.469414 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.469367 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b83871e-854a-4e91-8857-6c63753834db-openshift-service-ca" pod "5b83871e-854a-4e91-8857-6c63753834db" (UID: "5b83871e-854a-4e91-8857-6c63753834db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:40.470727 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.470700 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b83871e-854a-4e91-8857-6c63753834db-metrics-certs" pod "5b83871e-854a-4e91-8857-6c63753834db" (UID: "5b83871e-854a-4e91-8857-6c63753834db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:40.471169 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.471145 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b83871e-854a-4e91-8857-6c63753834db-maas-api-tls" pod "5b83871e-854a-4e91-8857-6c63753834db" (UID: "5b83871e-854a-4e91-8857-6c63753834db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:31:40.471576 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.471550 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84"] Oct 30 12:31:40.471686 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.471646 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b83871e-854a-4e91-8857-6c63753834db-kube-api-access-zd9lq" pod "5b83871e-854a-4e91-8857-6c63753834db" (UID: "5b83871e-854a-4e91-8857-6c63753834db"). InnerVolumeSpecName "kube-api-access-zd9lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:40.509389 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.509357 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:40.569157 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.569086 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2efc5e39-2e4f-4700-96d3-0b478cac830f-istiod-ca-cert\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.569157 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.569129 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm6p9\" (UniqueName: \"kubernetes.io/projected/2efc5e39-2e4f-4700-96d3-0b478cac830f-kube-api-access-vm6p9\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.569157 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.569147 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-token\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.569412 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.569281 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-workload-socket\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.569412 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.569315 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-workload-certs\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.569412 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.569337 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-envoy\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.569540 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.569473 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-data\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.569540 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.569503 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-podinfo\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.569540 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.569528 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-credential-socket\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.569719 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.569701 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd9lq\" (UniqueName: \"kubernetes.io/projected/5b83871e-854a-4e91-8857-6c63753834db-kube-api-access-zd9lq\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:40.569782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.569729 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b83871e-854a-4e91-8857-6c63753834db-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:40.569782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.569745 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b83871e-854a-4e91-8857-6c63753834db-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:40.569782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.569760 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b83871e-854a-4e91-8857-6c63753834db-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:40.670817 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.670774 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-data\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.672364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.670841 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-podinfo\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.672364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.670895 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-credential-socket\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.672364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.670974 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2efc5e39-2e4f-4700-96d3-0b478cac830f-istiod-ca-cert\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.672364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.671025 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm6p9\" (UniqueName: \"kubernetes.io/projected/2efc5e39-2e4f-4700-96d3-0b478cac830f-kube-api-access-vm6p9\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.672364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.671053 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-token\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.672364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.671110 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-workload-socket\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.672364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.671142 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-workload-certs\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.672364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.671173 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-envoy\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.672364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.671468 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-data\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.672364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.671643 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-credential-socket\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.672364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.671693 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-workload-socket\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.672364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.672127 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2efc5e39-2e4f-4700-96d3-0b478cac830f-istiod-ca-cert\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.672364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.672321 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-workload-certs\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.674196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.674178 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-podinfo\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.674512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.674497 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-envoy\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.695810 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.695783 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-token\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.700469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.700452 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm6p9\" (UniqueName: \"kubernetes.io/projected/2efc5e39-2e4f-4700-96d3-0b478cac830f-kube-api-access-vm6p9\") pod \"e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.766557 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.766515 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:40.867508 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.864772 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:40.982691 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.982646 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:40.992455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.991561 4413 generic.go:356] "Generic (PLEG): container finished" podID="5b83871e-854a-4e91-8857-6c63753834db" containerID="265a089eb19dcf450bd61fa1a8a432ac18198d3d49cc8dded812b0aebecc1a92" exitCode=0 Oct 30 12:31:40.992455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.991776 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" event={"ID":"5b83871e-854a-4e91-8857-6c63753834db","Type":"ContainerDied","Data":"265a089eb19dcf450bd61fa1a8a432ac18198d3d49cc8dded812b0aebecc1a92"} Oct 30 12:31:40.992455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.991874 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" event={"ID":"5b83871e-854a-4e91-8857-6c63753834db","Type":"ContainerDied","Data":"2a668064dd2a2e6409623e219efa8eb52f40fd76b8fd2dffce26d27066d5faa8"} Oct 30 12:31:40.992455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.991912 4413 scope.go:122] "RemoveContainer" containerID="265a089eb19dcf450bd61fa1a8a432ac18198d3d49cc8dded812b0aebecc1a92" Oct 30 12:31:40.992455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:40.992100 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr" Oct 30 12:31:41.009339 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:41.009312 4413 scope.go:122] "RemoveContainer" containerID="265a089eb19dcf450bd61fa1a8a432ac18198d3d49cc8dded812b0aebecc1a92" Oct 30 12:31:41.009665 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:31:41.009644 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"265a089eb19dcf450bd61fa1a8a432ac18198d3d49cc8dded812b0aebecc1a92\": container with ID starting with 265a089eb19dcf450bd61fa1a8a432ac18198d3d49cc8dded812b0aebecc1a92 not found: ID does not exist" containerID="265a089eb19dcf450bd61fa1a8a432ac18198d3d49cc8dded812b0aebecc1a92" Oct 30 12:31:41.009741 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:41.009674 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"265a089eb19dcf450bd61fa1a8a432ac18198d3d49cc8dded812b0aebecc1a92"} err="failed to get container status \"265a089eb19dcf450bd61fa1a8a432ac18198d3d49cc8dded812b0aebecc1a92\": rpc error: code = NotFound desc = could not find container \"265a089eb19dcf450bd61fa1a8a432ac18198d3d49cc8dded812b0aebecc1a92\": container with ID starting with 265a089eb19dcf450bd61fa1a8a432ac18198d3d49cc8dded812b0aebecc1a92 not found: ID does not exist" Oct 30 12:31:41.020124 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:41.020102 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78cbff45-e07e-4847-af31-667665133aeb" path="/var/lib/kubelet/pods/78cbff45-e07e-4847-af31-667665133aeb/volumes" Oct 30 12:31:41.045718 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:41.045685 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr"] Oct 30 12:31:41.049102 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:41.049084 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1e9675-556b686f99-fjfbr"] Oct 30 12:31:41.111432 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:41.111337 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:41.120077 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:41.120052 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84"] Oct 30 12:31:41.122743 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:31:41.122711 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2efc5e39_2e4f_4700_96d3_0b478cac830f.slice/crio-d5a800939366a0062e35cb0daf30ab82c3f0a3d9e67e0f897b931c84b729eaa7 WatchSource:0}: Error finding container d5a800939366a0062e35cb0daf30ab82c3f0a3d9e67e0f897b931c84b729eaa7: Status 404 returned error can't find the container with id d5a800939366a0062e35cb0daf30ab82c3f0a3d9e67e0f897b931c84b729eaa7 Oct 30 12:31:41.126147 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:41.126105 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:31:41.126289 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:41.126189 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:31:41.126289 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:41.126228 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:31:41.492732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:41.492686 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:42.001073 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:42.001037 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" event={"ID":"2efc5e39-2e4f-4700-96d3-0b478cac830f","Type":"ContainerStarted","Data":"db3e8b6e262d9fae95ea47f5f2210d5f9962b0bd0a67bff7ddd4140c529c6546"} Oct 30 12:31:42.001297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:42.001101 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" event={"ID":"2efc5e39-2e4f-4700-96d3-0b478cac830f","Type":"ContainerStarted","Data":"d5a800939366a0062e35cb0daf30ab82c3f0a3d9e67e0f897b931c84b729eaa7"} Oct 30 12:31:42.049131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:42.049081 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" podStartSLOduration=2.049066537 podStartE2EDuration="2.049066537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:42.047127614 +0000 UTC m=+2255.156970495" watchObservedRunningTime="2026-10-30 12:31:42.049066537 +0000 UTC m=+2255.158909418" Oct 30 12:31:42.767677 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:42.767640 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:42.773713 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:42.773620 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:42.774578 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:42.774558 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:43.009220 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.009189 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:43.010195 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.010174 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:31:43.020316 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.020241 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b83871e-854a-4e91-8857-6c63753834db" path="/var/lib/kubelet/pods/5b83871e-854a-4e91-8857-6c63753834db/volumes" Oct 30 12:31:43.634669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.634645 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:31:43.709378 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.709294 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-envoy\") pod \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " Oct 30 12:31:43.709541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.709385 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istiod-ca-cert\") pod \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " Oct 30 12:31:43.709541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.709529 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-workload-certs\") pod \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " Oct 30 12:31:43.709672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.709594 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0250c9a9-b527-47e2-8d57-f221fcbd8e19-kube-api-access-m7kq4\" (UniqueName: \"kubernetes.io/projected/0250c9a9-b527-47e2-8d57-f221fcbd8e19-kube-api-access-m7kq4\") pod \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " Oct 30 12:31:43.709672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.709643 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-workload-socket\") pod \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " Oct 30 12:31:43.709793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.709771 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-data\") pod \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " Oct 30 12:31:43.709793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.709775 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-workload-certs" pod "0250c9a9-b527-47e2-8d57-f221fcbd8e19" (UID: "0250c9a9-b527-47e2-8d57-f221fcbd8e19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:43.709895 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.709798 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istiod-ca-cert" pod "0250c9a9-b527-47e2-8d57-f221fcbd8e19" (UID: "0250c9a9-b527-47e2-8d57-f221fcbd8e19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:31:43.709895 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.709834 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-credential-socket\") pod \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " Oct 30 12:31:43.709895 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.709867 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-token\" (UniqueName: \"kubernetes.io/projected/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-token\") pod \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " Oct 30 12:31:43.710049 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.709916 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-podinfo\") pod \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\" (UID: \"0250c9a9-b527-47e2-8d57-f221fcbd8e19\") " Oct 30 12:31:43.710049 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.709989 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-workload-socket" pod "0250c9a9-b527-47e2-8d57-f221fcbd8e19" (UID: "0250c9a9-b527-47e2-8d57-f221fcbd8e19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:43.710361 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.710130 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-data" pod "0250c9a9-b527-47e2-8d57-f221fcbd8e19" (UID: "0250c9a9-b527-47e2-8d57-f221fcbd8e19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:43.710361 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.710204 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-credential-socket" pod "0250c9a9-b527-47e2-8d57-f221fcbd8e19" (UID: "0250c9a9-b527-47e2-8d57-f221fcbd8e19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:43.710531 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.710511 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:43.710594 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.710537 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:43.710594 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.710549 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:43.710594 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.710564 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:43.710594 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.710580 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:43.711873 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.711849 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-envoy" pod "0250c9a9-b527-47e2-8d57-f221fcbd8e19" (UID: "0250c9a9-b527-47e2-8d57-f221fcbd8e19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:31:43.712524 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.712495 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-podinfo" pod "0250c9a9-b527-47e2-8d57-f221fcbd8e19" (UID: "0250c9a9-b527-47e2-8d57-f221fcbd8e19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:31:43.712918 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.712900 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-token" pod "0250c9a9-b527-47e2-8d57-f221fcbd8e19" (UID: "0250c9a9-b527-47e2-8d57-f221fcbd8e19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:43.713083 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.713051 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0250c9a9-b527-47e2-8d57-f221fcbd8e19-kube-api-access-m7kq4" pod "0250c9a9-b527-47e2-8d57-f221fcbd8e19" (UID: "0250c9a9-b527-47e2-8d57-f221fcbd8e19"). InnerVolumeSpecName "kube-api-access-m7kq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:31:43.811817 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.811789 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7kq4\" (UniqueName: \"kubernetes.io/projected/0250c9a9-b527-47e2-8d57-f221fcbd8e19-kube-api-access-m7kq4\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:43.811817 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.811813 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:43.811817 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.811823 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:43.812243 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:43.811830 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0250c9a9-b527-47e2-8d57-f221fcbd8e19-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:31:44.018081 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:44.017992 4413 generic.go:356] "Generic (PLEG): container finished" podID="0250c9a9-b527-47e2-8d57-f221fcbd8e19" containerID="7d4e0571053faf2806ac90492208c9dbc0e2f8ae4e384ac966355c52be042886" exitCode=0 Oct 30 12:31:44.018081 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:44.018035 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" Oct 30 12:31:44.018081 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:44.018074 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" event={"ID":"0250c9a9-b527-47e2-8d57-f221fcbd8e19","Type":"ContainerDied","Data":"7d4e0571053faf2806ac90492208c9dbc0e2f8ae4e384ac966355c52be042886"} Oct 30 12:31:44.018386 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:44.018121 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj" event={"ID":"0250c9a9-b527-47e2-8d57-f221fcbd8e19","Type":"ContainerDied","Data":"a3e44c074cebb6ebb836fb5f43eea0b6fbb8f63f29b4f843b1cbb0cdcef7a47f"} Oct 30 12:31:44.018386 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:44.018149 4413 scope.go:122] "RemoveContainer" containerID="7d4e0571053faf2806ac90492208c9dbc0e2f8ae4e384ac966355c52be042886" Oct 30 12:31:44.028608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:44.028591 4413 scope.go:122] "RemoveContainer" containerID="6f2735db2e680fe4fc391bee550fba4ad193823dbad37128da8fe2bbefc96cb2" Oct 30 12:31:44.037443 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:44.037425 4413 scope.go:122] "RemoveContainer" containerID="7d4e0571053faf2806ac90492208c9dbc0e2f8ae4e384ac966355c52be042886" Oct 30 12:31:44.037725 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:31:44.037704 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d4e0571053faf2806ac90492208c9dbc0e2f8ae4e384ac966355c52be042886\": container with ID starting with 7d4e0571053faf2806ac90492208c9dbc0e2f8ae4e384ac966355c52be042886 not found: ID does not exist" containerID="7d4e0571053faf2806ac90492208c9dbc0e2f8ae4e384ac966355c52be042886" Oct 30 12:31:44.037802 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:44.037738 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d4e0571053faf2806ac90492208c9dbc0e2f8ae4e384ac966355c52be042886"} err="failed to get container status \"7d4e0571053faf2806ac90492208c9dbc0e2f8ae4e384ac966355c52be042886\": rpc error: code = NotFound desc = could not find container \"7d4e0571053faf2806ac90492208c9dbc0e2f8ae4e384ac966355c52be042886\": container with ID starting with 7d4e0571053faf2806ac90492208c9dbc0e2f8ae4e384ac966355c52be042886 not found: ID does not exist" Oct 30 12:31:44.037802 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:44.037763 4413 scope.go:122] "RemoveContainer" containerID="6f2735db2e680fe4fc391bee550fba4ad193823dbad37128da8fe2bbefc96cb2" Oct 30 12:31:44.038037 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:31:44.038018 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2735db2e680fe4fc391bee550fba4ad193823dbad37128da8fe2bbefc96cb2\": container with ID starting with 6f2735db2e680fe4fc391bee550fba4ad193823dbad37128da8fe2bbefc96cb2 not found: ID does not exist" containerID="6f2735db2e680fe4fc391bee550fba4ad193823dbad37128da8fe2bbefc96cb2" Oct 30 12:31:44.038078 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:44.038044 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2735db2e680fe4fc391bee550fba4ad193823dbad37128da8fe2bbefc96cb2"} err="failed to get container status \"6f2735db2e680fe4fc391bee550fba4ad193823dbad37128da8fe2bbefc96cb2\": rpc error: code = NotFound desc = could not find container \"6f2735db2e680fe4fc391bee550fba4ad193823dbad37128da8fe2bbefc96cb2\": container with ID starting with 6f2735db2e680fe4fc391bee550fba4ad193823dbad37128da8fe2bbefc96cb2 not found: ID does not exist" Oct 30 12:31:44.070771 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:44.070751 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj"] Oct 30 12:31:44.082803 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:44.082772 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-7f78d4-openshift-default-7b89b6d6b9-7rcmj"] Oct 30 12:31:44.100734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:44.100705 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:44.218875 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:44.218845 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:44.452236 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:44.452206 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:44.565115 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:44.565088 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:44.798367 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:44.794895 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:45.024976 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:45.024936 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0250c9a9-b527-47e2-8d57-f221fcbd8e19" path="/var/lib/kubelet/pods/0250c9a9-b527-47e2-8d57-f221fcbd8e19/volumes" Oct 30 12:31:45.340729 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:45.340695 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:45.456735 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:45.456702 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:46.319109 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.319072 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:46.433834 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.433802 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:46.488732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.488702 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x"] Oct 30 12:31:46.489233 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.489221 4413 cpu_manager.go:414] "containerMap: removing container" podUID="0250c9a9-b527-47e2-8d57-f221fcbd8e19" containerName="istio-proxy" Oct 30 12:31:46.489301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.489235 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0250c9a9-b527-47e2-8d57-f221fcbd8e19" containerName="istio-proxy" Oct 30 12:31:46.489373 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.489363 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="0250c9a9-b527-47e2-8d57-f221fcbd8e19" containerName="istio-proxy" Oct 30 12:31:46.492895 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.492877 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x" Oct 30 12:31:46.507552 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.507527 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-0172c6\"" Oct 30 12:31:46.519407 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.519380 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-0172c6-dockercfg-p26wv\"" Oct 30 12:31:46.522165 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.522145 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x"] Oct 30 12:31:46.550593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.550565 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x\" (UID: \"95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x" Oct 30 12:31:46.550731 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.550675 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb6qh\" (UniqueName: \"kubernetes.io/projected/95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6-kube-api-access-cb6qh\") pod \"payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x\" (UID: \"95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x" Oct 30 12:31:46.592755 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.592639 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm"] Oct 30 12:31:46.596691 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.596655 4413 cpu_manager.go:414] "containerMap: removing container" podUID="0250c9a9-b527-47e2-8d57-f221fcbd8e19" containerName="istio-proxy" Oct 30 12:31:46.596930 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.596912 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0250c9a9-b527-47e2-8d57-f221fcbd8e19" containerName="istio-proxy" Oct 30 12:31:46.597214 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.597198 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="0250c9a9-b527-47e2-8d57-f221fcbd8e19" containerName="istio-proxy" Oct 30 12:31:46.600606 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.600578 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm" Oct 30 12:31:46.622280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.618946 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm"] Oct 30 12:31:46.658502 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.658465 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x\" (UID: \"95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x" Oct 30 12:31:46.658716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.658573 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm\" (UID: \"b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm" Oct 30 12:31:46.658797 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.658751 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb6qh\" (UniqueName: \"kubernetes.io/projected/95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6-kube-api-access-cb6qh\") pod \"payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x\" (UID: \"95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x" Oct 30 12:31:46.658876 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.658861 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2v59\" (UniqueName: \"kubernetes.io/projected/b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9-kube-api-access-m2v59\") pod \"payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm\" (UID: \"b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm" Oct 30 12:31:46.659584 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.659559 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x\" (UID: \"95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x" Oct 30 12:31:46.682840 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.682811 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb6qh\" (UniqueName: \"kubernetes.io/projected/95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6-kube-api-access-cb6qh\") pod \"payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x\" (UID: \"95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x" Oct 30 12:31:46.760386 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.760347 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2v59\" (UniqueName: \"kubernetes.io/projected/b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9-kube-api-access-m2v59\") pod \"payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm\" (UID: \"b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm" Oct 30 12:31:46.760595 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.760486 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm\" (UID: \"b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm" Oct 30 12:31:46.761193 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.761175 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm\" (UID: \"b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm" Oct 30 12:31:46.785467 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.785429 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2v59\" (UniqueName: \"kubernetes.io/projected/b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9-kube-api-access-m2v59\") pod \"payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm\" (UID: \"b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm" Oct 30 12:31:46.805420 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.805397 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x" Oct 30 12:31:46.835383 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.835361 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:46.941491 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:46.941450 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm" Oct 30 12:31:47.132744 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:47.132713 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x"] Oct 30 12:31:47.133306 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:31:47.133278 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f0f90e_ba2f_46e4_8788_cc6a9c9d47e6.slice/crio-de2b1fff7637f2a373e90cdc629c95fd071d13e1d1c08b1b0130911b3ab6397a WatchSource:0}: Error finding container de2b1fff7637f2a373e90cdc629c95fd071d13e1d1c08b1b0130911b3ab6397a: Status 404 returned error can't find the container with id de2b1fff7637f2a373e90cdc629c95fd071d13e1d1c08b1b0130911b3ab6397a Oct 30 12:31:47.280981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:47.280953 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm"] Oct 30 12:31:47.282103 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:31:47.282070 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b0d9a1_da2a_46ee_a016_0993d8fc4cb9.slice/crio-c59a9661582e2558b8247947b980f050e8218caec913590e26f6f308e3a34ffa WatchSource:0}: Error finding container c59a9661582e2558b8247947b980f050e8218caec913590e26f6f308e3a34ffa: Status 404 returned error can't find the container with id c59a9661582e2558b8247947b980f050e8218caec913590e26f6f308e3a34ffa Oct 30 12:31:48.050968 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:48.050928 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm" event={"ID":"b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9","Type":"ContainerStarted","Data":"e20e52cc3283bdf98c2b9dc0b7ac87794fc64e17da5fb71608461fec2290d36d"} Oct 30 12:31:48.051491 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:48.050990 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm" event={"ID":"b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9","Type":"ContainerStarted","Data":"c59a9661582e2558b8247947b980f050e8218caec913590e26f6f308e3a34ffa"} Oct 30 12:31:48.051491 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:48.051023 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm" Oct 30 12:31:48.052555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:48.052535 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x" event={"ID":"95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6","Type":"ContainerStarted","Data":"9ed159526f223e57dc49588a7cb4976e8e84b0823cf4284e812106a53206d07b"} Oct 30 12:31:48.052674 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:48.052572 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x" event={"ID":"95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6","Type":"ContainerStarted","Data":"de2b1fff7637f2a373e90cdc629c95fd071d13e1d1c08b1b0130911b3ab6397a"} Oct 30 12:31:48.052674 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:48.052598 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x" Oct 30 12:31:48.064937 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:48.064892 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:48.091062 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:48.091012 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm" podStartSLOduration=2.09098842 podStartE2EDuration="2.09098842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:48.090086107 +0000 UTC m=+2261.199928992" watchObservedRunningTime="2026-10-30 12:31:48.09098842 +0000 UTC m=+2261.200831301" Oct 30 12:31:48.131685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:48.131629 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x" podStartSLOduration=2.13161108 podStartE2EDuration="2.13161108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:31:48.129215992 +0000 UTC m=+2261.239058876" watchObservedRunningTime="2026-10-30 12:31:48.13161108 +0000 UTC m=+2261.241453965" Oct 30 12:31:48.176058 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:48.176024 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:48.433725 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:48.433673 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:48.790432 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:48.790346 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:50.710176 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:50.710132 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:52.788467 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:52.788432 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:54.344947 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:54.344916 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:56.043054 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:56.043016 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:57.769205 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:57.769155 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:58.199610 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:58.199575 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:31:59.060734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:59.060708 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm" Oct 30 12:31:59.061120 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:59.060877 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x" Oct 30 12:31:59.459439 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:31:59.459407 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:01.289052 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.289021 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd"] Oct 30 12:32:01.310083 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.308860 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" Oct 30 12:32:01.328697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.326777 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-0172c6-serving-cert\"" Oct 30 12:32:01.377275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.373027 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64cfb9ee-656e-4d1b-af81-1a73742935b7-metrics-certs\") pod \"maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd\" (UID: \"64cfb9ee-656e-4d1b-af81-1a73742935b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" Oct 30 12:32:01.377275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.373124 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64cfb9ee-656e-4d1b-af81-1a73742935b7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd\" (UID: \"64cfb9ee-656e-4d1b-af81-1a73742935b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" Oct 30 12:32:01.377275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.373177 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s58qv\" (UniqueName: \"kubernetes.io/projected/64cfb9ee-656e-4d1b-af81-1a73742935b7-kube-api-access-s58qv\") pod \"maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd\" (UID: \"64cfb9ee-656e-4d1b-af81-1a73742935b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" Oct 30 12:32:01.377275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.373291 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64cfb9ee-656e-4d1b-af81-1a73742935b7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd\" (UID: \"64cfb9ee-656e-4d1b-af81-1a73742935b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" Oct 30 12:32:01.378736 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.378679 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd"] Oct 30 12:32:01.415430 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.415380 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:01.474658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.473969 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64cfb9ee-656e-4d1b-af81-1a73742935b7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd\" (UID: \"64cfb9ee-656e-4d1b-af81-1a73742935b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" Oct 30 12:32:01.474658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.474053 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64cfb9ee-656e-4d1b-af81-1a73742935b7-metrics-certs\") pod \"maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd\" (UID: \"64cfb9ee-656e-4d1b-af81-1a73742935b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" Oct 30 12:32:01.474658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.474107 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64cfb9ee-656e-4d1b-af81-1a73742935b7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd\" (UID: \"64cfb9ee-656e-4d1b-af81-1a73742935b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" Oct 30 12:32:01.474658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.474138 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s58qv\" (UniqueName: \"kubernetes.io/projected/64cfb9ee-656e-4d1b-af81-1a73742935b7-kube-api-access-s58qv\") pod \"maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd\" (UID: \"64cfb9ee-656e-4d1b-af81-1a73742935b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" Oct 30 12:32:01.475532 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.475509 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64cfb9ee-656e-4d1b-af81-1a73742935b7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd\" (UID: \"64cfb9ee-656e-4d1b-af81-1a73742935b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" Oct 30 12:32:01.477810 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.477784 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64cfb9ee-656e-4d1b-af81-1a73742935b7-metrics-certs\") pod \"maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd\" (UID: \"64cfb9ee-656e-4d1b-af81-1a73742935b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" Oct 30 12:32:01.477982 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.477963 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64cfb9ee-656e-4d1b-af81-1a73742935b7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd\" (UID: \"64cfb9ee-656e-4d1b-af81-1a73742935b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" Oct 30 12:32:01.499655 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.499625 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s58qv\" (UniqueName: \"kubernetes.io/projected/64cfb9ee-656e-4d1b-af81-1a73742935b7-kube-api-access-s58qv\") pod \"maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd\" (UID: \"64cfb9ee-656e-4d1b-af81-1a73742935b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" Oct 30 12:32:01.560486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.560393 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:01.646437 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.646399 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" Oct 30 12:32:01.999922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:01.999898 4413 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:32:02.002593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:02.002566 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd"] Oct 30 12:32:02.176692 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:02.176658 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" event={"ID":"64cfb9ee-656e-4d1b-af81-1a73742935b7","Type":"ContainerStarted","Data":"46f0613a36e5bb74f07a2fb8ef3e68e6935d3523213559a38e85cde7880f9672"} Oct 30 12:32:03.143628 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:03.143588 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:03.198709 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:03.197940 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" event={"ID":"64cfb9ee-656e-4d1b-af81-1a73742935b7","Type":"ContainerStarted","Data":"5f2ec7460944df933c968edf735d7d3fb9a89436f6857ec294d5e46ddfa5a27b"} Oct 30 12:32:03.208271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:03.199506 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" Oct 30 12:32:03.242151 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:03.242094 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" podStartSLOduration=1.67388113 podStartE2EDuration="2.242079187s" totalImagesPullingTime="568.198057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:03.240789348 +0000 UTC m=+2276.350632242" watchObservedRunningTime="2026-10-30 12:32:03.242079187 +0000 UTC m=+2276.351922065" Oct 30 12:32:04.739825 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:04.739780 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:05.814156 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:32:05.814121 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:32:06.379395 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:06.379343 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:07.670612 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:07.670581 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn"] Oct 30 12:32:07.671130 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:07.670831 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" podUID="7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c" containerName="maas-api" containerID="cri-o://9609a63cded91a8ce2664f97002640a1b588183459a54a2bbd6f1001029b93be" gracePeriod=30 Oct 30 12:32:07.737393 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:07.737355 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:07.920276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:07.920223 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:07.990007 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:07.989971 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" Oct 30 12:32:08.047846 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.047817 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:08.052064 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.052039 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-maas-api-tls\") pod \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\" (UID: \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\") " Oct 30 12:32:08.052140 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.052086 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-kube-api-access-ng6cf\" (UniqueName: \"kubernetes.io/projected/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-kube-api-access-ng6cf\") pod \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\" (UID: \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\") " Oct 30 12:32:08.052140 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.052124 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-openshift-service-ca\") pod \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\" (UID: \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\") " Oct 30 12:32:08.052282 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.052266 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-metrics-certs\") pod \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\" (UID: \"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c\") " Oct 30 12:32:08.052654 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.052619 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-openshift-service-ca" pod "7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c" (UID: "7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:08.053396 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.052889 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:32:08.054503 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.054477 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-kube-api-access-ng6cf" pod "7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c" (UID: "7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c"). InnerVolumeSpecName "kube-api-access-ng6cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:08.055109 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.055079 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-metrics-certs" pod "7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c" (UID: "7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:32:08.055327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.055307 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-maas-api-tls" pod "7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c" (UID: "7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:32:08.154441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.154408 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:32:08.154441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.154437 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng6cf\" (UniqueName: \"kubernetes.io/projected/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-kube-api-access-ng6cf\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:32:08.154441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.154447 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:32:08.235342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.235220 4413 generic.go:356] "Generic (PLEG): container finished" podID="7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c" containerID="9609a63cded91a8ce2664f97002640a1b588183459a54a2bbd6f1001029b93be" exitCode=0 Oct 30 12:32:08.235342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.235292 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" Oct 30 12:32:08.235342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.235290 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" event={"ID":"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c","Type":"ContainerDied","Data":"9609a63cded91a8ce2664f97002640a1b588183459a54a2bbd6f1001029b93be"} Oct 30 12:32:08.235655 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.235352 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn" event={"ID":"7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c","Type":"ContainerDied","Data":"7ad75107079cc4d607e8c5c1b122e0ad701b1faf46076b38ec0491b39fa465e8"} Oct 30 12:32:08.235655 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.235385 4413 scope.go:122] "RemoveContainer" containerID="9609a63cded91a8ce2664f97002640a1b588183459a54a2bbd6f1001029b93be" Oct 30 12:32:08.252773 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.252742 4413 scope.go:122] "RemoveContainer" containerID="9609a63cded91a8ce2664f97002640a1b588183459a54a2bbd6f1001029b93be" Oct 30 12:32:08.254343 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:32:08.254311 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9609a63cded91a8ce2664f97002640a1b588183459a54a2bbd6f1001029b93be\": container with ID starting with 9609a63cded91a8ce2664f97002640a1b588183459a54a2bbd6f1001029b93be not found: ID does not exist" containerID="9609a63cded91a8ce2664f97002640a1b588183459a54a2bbd6f1001029b93be" Oct 30 12:32:08.254459 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.254348 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9609a63cded91a8ce2664f97002640a1b588183459a54a2bbd6f1001029b93be"} err="failed to get container status \"9609a63cded91a8ce2664f97002640a1b588183459a54a2bbd6f1001029b93be\": rpc error: code = NotFound desc = could not find container \"9609a63cded91a8ce2664f97002640a1b588183459a54a2bbd6f1001029b93be\": container with ID starting with 9609a63cded91a8ce2664f97002640a1b588183459a54a2bbd6f1001029b93be not found: ID does not exist" Oct 30 12:32:08.270783 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.270745 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:08.298286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.291603 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn"] Oct 30 12:32:08.298286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.294451 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c57af-6bfdf6bf8f-hqpnn"] Oct 30 12:32:08.403203 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.403151 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:08.521452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.521363 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:08.939202 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:08.939168 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:09.020780 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:09.020738 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c" path="/var/lib/kubelet/pods/7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c/volumes" Oct 30 12:32:09.210015 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:09.209948 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" Oct 30 12:32:10.141962 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:10.141934 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:11.689922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:11.689890 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:13.500070 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:13.500039 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:15.054948 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:15.054914 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:15.270856 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:15.270821 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:15.391929 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:15.391895 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:15.586518 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:15.586484 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw"] Oct 30 12:32:15.586752 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:15.586723 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" podUID="2cb74db0-df6e-471c-bce4-7de34b2a33e3" containerName="istio-proxy" containerID="cri-o://55f66ad0552618147760f065b20735848b774f655c64da2e36479e18669a6de3" gracePeriod=30 Oct 30 12:32:15.626153 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:15.626117 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:16.652294 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:16.652239 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:17.237978 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.237945 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:17.665694 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.665663 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf"] Oct 30 12:32:17.666530 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.666510 4413 cpu_manager.go:414] "containerMap: removing container" podUID="7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c" containerName="maas-api" Oct 30 12:32:17.666647 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.666531 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c" containerName="maas-api" Oct 30 12:32:17.666703 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.666677 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bdc2d24-ac84-4bc0-9689-6fa705ccbb2c" containerName="maas-api" Oct 30 12:32:17.671553 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.671534 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.686512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.686494 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-a124ab-openshift-default-dockercfg-hqzcf\"" Oct 30 12:32:17.701871 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.701843 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf"] Oct 30 12:32:17.725660 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.725625 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:17.796430 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.796397 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-envoy\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.796584 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.796470 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-podinfo\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.796629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.796587 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-workload-socket\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.796683 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.796636 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istiod-ca-cert\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.796720 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.796695 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-workload-certs\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.796781 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.796767 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk5v2\" (UniqueName: \"kubernetes.io/projected/db669f7d-1af0-4f6d-9a65-f3295751ceb8-kube-api-access-bk5v2\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.796838 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.796818 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-data\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.796930 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.796918 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-token\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.796969 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.796955 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-credential-socket\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.859603 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.859565 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:17.914452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.908072 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk5v2\" (UniqueName: \"kubernetes.io/projected/db669f7d-1af0-4f6d-9a65-f3295751ceb8-kube-api-access-bk5v2\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.914452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.908127 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-data\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.914452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.908206 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-token\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.914452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.908271 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-credential-socket\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.914452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.908312 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-envoy\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.914452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.908355 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-podinfo\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.914452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.908447 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-workload-socket\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.914452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.908501 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istiod-ca-cert\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.914452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.908539 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-workload-certs\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.914452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.909062 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-workload-certs\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.914452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.909656 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-data\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.914452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.910153 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-credential-socket\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.937337 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.926029 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-workload-socket\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.937337 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.936793 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-podinfo\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.938213 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.938178 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istiod-ca-cert\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.942146 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.942091 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-token\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.942656 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.942610 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-envoy\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.945447 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.945400 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk5v2\" (UniqueName: \"kubernetes.io/projected/db669f7d-1af0-4f6d-9a65-f3295751ceb8-kube-api-access-bk5v2\") pod \"e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:17.987332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:17.987297 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:18.307919 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:18.307893 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf"] Oct 30 12:32:18.308974 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:32:18.308942 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb669f7d_1af0_4f6d_9a65_f3295751ceb8.slice/crio-d67e1e3cc2051b0b1f6165318b491eb682c917d1ce63c55f8574b06eb2862876 WatchSource:0}: Error finding container d67e1e3cc2051b0b1f6165318b491eb682c917d1ce63c55f8574b06eb2862876: Status 404 returned error can't find the container with id d67e1e3cc2051b0b1f6165318b491eb682c917d1ce63c55f8574b06eb2862876 Oct 30 12:32:18.312074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:18.312036 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:32:18.312148 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:18.312120 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:32:18.312193 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:18.312163 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:32:18.321978 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:18.321940 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" event={"ID":"db669f7d-1af0-4f6d-9a65-f3295751ceb8","Type":"ContainerStarted","Data":"d67e1e3cc2051b0b1f6165318b491eb682c917d1ce63c55f8574b06eb2862876"} Oct 30 12:32:18.365968 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:18.365937 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:19.343813 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:19.343721 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" event={"ID":"db669f7d-1af0-4f6d-9a65-f3295751ceb8","Type":"ContainerStarted","Data":"9c5df2ade0ea8f11cc2fbb19bdc805d5bff57ab62982e9957090a760284dc47f"} Oct 30 12:32:19.405587 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:19.405525 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" podStartSLOduration=2.405503994 podStartE2EDuration="2.405503994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:19.404739813 +0000 UTC m=+2292.514582709" watchObservedRunningTime="2026-10-30 12:32:19.405503994 +0000 UTC m=+2292.515346876" Oct 30 12:32:19.944158 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:19.944123 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:19.999416 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:19.999374 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp"] Oct 30 12:32:20.009420 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.009387 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:20.009614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.009505 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.073612 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.067362 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:20.423274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.417385 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.423274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.417457 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jb5j\" (UniqueName: \"kubernetes.io/projected/fdd40382-d780-4cab-941f-b2ef96ac86a0-kube-api-access-9jb5j\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.423274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.417523 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.423274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.417675 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd40382-d780-4cab-941f-b2ef96ac86a0-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.423274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.417739 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.423274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.417821 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.423274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.417873 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.423274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.417908 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.423274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.418063 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.580326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.467431 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp"] Oct 30 12:32:20.580326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.524305 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.580326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.524387 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.580326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.556384 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jb5j\" (UniqueName: \"kubernetes.io/projected/fdd40382-d780-4cab-941f-b2ef96ac86a0-kube-api-access-9jb5j\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.580326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.556461 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.580326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.556628 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd40382-d780-4cab-941f-b2ef96ac86a0-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.580326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.556685 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.580326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.556768 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.580326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.556823 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.580326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.556862 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.580326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.557540 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.580326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.565378 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd40382-d780-4cab-941f-b2ef96ac86a0-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.580326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.566034 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.580326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.574509 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.580326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.575402 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.580326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.575668 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.651276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.641188 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.700345 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.669958 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:20.806335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.798347 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.806335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.798856 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jb5j\" (UniqueName: \"kubernetes.io/projected/fdd40382-d780-4cab-941f-b2ef96ac86a0-kube-api-access-9jb5j\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:20.815804 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.813373 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:20.817883 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.816365 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:32:20.869870 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:20.869060 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:21.166686 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.166645 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:21.181019 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.180992 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:32:21.261835 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.261753 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-token\" (UniqueName: \"kubernetes.io/projected/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-token\") pod \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " Oct 30 12:32:21.262003 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.261854 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-data\") pod \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " Oct 30 12:32:21.262003 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.261896 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-credential-socket\") pod \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " Oct 30 12:32:21.262003 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.261924 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb74db0-df6e-471c-bce4-7de34b2a33e3-kube-api-access-4qq9h\" (UniqueName: \"kubernetes.io/projected/2cb74db0-df6e-471c-bce4-7de34b2a33e3-kube-api-access-4qq9h\") pod \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " Oct 30 12:32:21.262186 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.262026 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-envoy\") pod \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " Oct 30 12:32:21.262186 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.262079 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-podinfo\") pod \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " Oct 30 12:32:21.262186 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.262108 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-workload-certs\") pod \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " Oct 30 12:32:21.262186 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.262172 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-workload-socket\") pod \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " Oct 30 12:32:21.262432 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.262211 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istiod-ca-cert\") pod \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\" (UID: \"2cb74db0-df6e-471c-bce4-7de34b2a33e3\") " Oct 30 12:32:21.262432 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.262333 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-credential-socket" pod "2cb74db0-df6e-471c-bce4-7de34b2a33e3" (UID: "2cb74db0-df6e-471c-bce4-7de34b2a33e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:21.262607 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.262455 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-data" pod "2cb74db0-df6e-471c-bce4-7de34b2a33e3" (UID: "2cb74db0-df6e-471c-bce4-7de34b2a33e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:21.262877 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.262850 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istiod-ca-cert" pod "2cb74db0-df6e-471c-bce4-7de34b2a33e3" (UID: "2cb74db0-df6e-471c-bce4-7de34b2a33e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:32:21.262948 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.262879 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:32:21.262948 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.262902 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:32:21.263113 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.263093 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-workload-socket" pod "2cb74db0-df6e-471c-bce4-7de34b2a33e3" (UID: "2cb74db0-df6e-471c-bce4-7de34b2a33e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:21.263227 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.263140 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-workload-certs" pod "2cb74db0-df6e-471c-bce4-7de34b2a33e3" (UID: "2cb74db0-df6e-471c-bce4-7de34b2a33e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:21.264963 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.264930 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-token" pod "2cb74db0-df6e-471c-bce4-7de34b2a33e3" (UID: "2cb74db0-df6e-471c-bce4-7de34b2a33e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:21.265071 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.265023 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-envoy" pod "2cb74db0-df6e-471c-bce4-7de34b2a33e3" (UID: "2cb74db0-df6e-471c-bce4-7de34b2a33e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:32:21.265400 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.265383 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-podinfo" pod "2cb74db0-df6e-471c-bce4-7de34b2a33e3" (UID: "2cb74db0-df6e-471c-bce4-7de34b2a33e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:32:21.265788 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.265774 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb74db0-df6e-471c-bce4-7de34b2a33e3-kube-api-access-4qq9h" pod "2cb74db0-df6e-471c-bce4-7de34b2a33e3" (UID: "2cb74db0-df6e-471c-bce4-7de34b2a33e3"). InnerVolumeSpecName "kube-api-access-4qq9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:32:21.341948 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.341921 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp"] Oct 30 12:32:21.343398 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:32:21.343368 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd40382_d780_4cab_941f_b2ef96ac86a0.slice/crio-d45926f41cb448514425256f23ca46400cd395fcd267c417a53874d24c824de4 WatchSource:0}: Error finding container d45926f41cb448514425256f23ca46400cd395fcd267c417a53874d24c824de4: Status 404 returned error can't find the container with id d45926f41cb448514425256f23ca46400cd395fcd267c417a53874d24c824de4 Oct 30 12:32:21.345843 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.345811 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:32:21.345917 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.345875 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:32:21.345917 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.345904 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:32:21.364576 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.364553 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qq9h\" (UniqueName: \"kubernetes.io/projected/2cb74db0-df6e-471c-bce4-7de34b2a33e3-kube-api-access-4qq9h\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:32:21.364666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.364580 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:32:21.364666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.364595 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:32:21.364666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.364607 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:32:21.364666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.364620 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb74db0-df6e-471c-bce4-7de34b2a33e3-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:32:21.364666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.364632 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:32:21.364666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.364644 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb74db0-df6e-471c-bce4-7de34b2a33e3-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:32:21.451579 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.451551 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn"] Oct 30 12:32:21.452326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.452306 4413 cpu_manager.go:414] "containerMap: removing container" podUID="2cb74db0-df6e-471c-bce4-7de34b2a33e3" containerName="istio-proxy" Oct 30 12:32:21.452426 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.452326 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb74db0-df6e-471c-bce4-7de34b2a33e3" containerName="istio-proxy" Oct 30 12:32:21.452426 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.452355 4413 cpu_manager.go:414] "containerMap: removing container" podUID="2cb74db0-df6e-471c-bce4-7de34b2a33e3" containerName="istio-proxy" Oct 30 12:32:21.452426 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.452363 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb74db0-df6e-471c-bce4-7de34b2a33e3" containerName="istio-proxy" Oct 30 12:32:21.452580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.452509 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb74db0-df6e-471c-bce4-7de34b2a33e3" containerName="istio-proxy" Oct 30 12:32:21.452580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.452534 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb74db0-df6e-471c-bce4-7de34b2a33e3" containerName="istio-proxy" Oct 30 12:32:21.456240 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.456220 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.479271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.479230 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn"] Oct 30 12:32:21.567527 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.567450 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5j74\" (UniqueName: \"kubernetes.io/projected/8b50ec75-3d82-436e-914c-0d667dbeb3ae-kube-api-access-z5j74\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.567527 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.567499 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.567753 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.567538 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.567753 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.567567 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.567753 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.567599 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.567753 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.567619 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.567753 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.567649 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.567753 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.567670 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.567753 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.567714 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.661658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.661625 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw"] Oct 30 12:32:21.666906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.666878 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.669069 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.669044 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5j74\" (UniqueName: \"kubernetes.io/projected/8b50ec75-3d82-436e-914c-0d667dbeb3ae-kube-api-access-z5j74\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.669268 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.669233 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.669786 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.669767 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.669928 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.669915 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.670060 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.670048 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.670159 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.670147 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.670309 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.670295 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.670415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.670401 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.670557 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.670544 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.671062 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.671042 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-workload-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.671176 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.669711 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-workload-certs\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.671826 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.671806 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istiod-ca-cert\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.673164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.672803 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-credential-socket\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.673790 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.673769 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-data\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.686148 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.685871 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-2461e8-openshift-default-dockercfg-b5k6f\"" Oct 30 12:32:21.731199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.731074 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5j74\" (UniqueName: \"kubernetes.io/projected/8b50ec75-3d82-436e-914c-0d667dbeb3ae-kube-api-access-z5j74\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.731199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.731084 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw"] Oct 30 12:32:21.731965 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.731909 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-podinfo\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.738403 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.738368 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-envoy\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.738549 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.738531 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-token\") pod \"e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.744473 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.744405 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:21.772480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.772446 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkxhj\" (UniqueName: \"kubernetes.io/projected/c04e3112-1c05-40db-a2ff-659b04c79ffa-kube-api-access-dkxhj\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.772643 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.772487 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04e3112-1c05-40db-a2ff-659b04c79ffa-istiod-ca-cert\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.772724 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.772658 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-token\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.772724 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.772701 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-podinfo\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.772851 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.772764 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-workload-certs\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.772937 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.772916 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-envoy\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.772988 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.772948 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-workload-socket\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.772988 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.772968 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-credential-socket\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.773092 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.773023 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-data\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.781447 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.781382 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:21.852504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.852400 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" event={"ID":"fdd40382-d780-4cab-941f-b2ef96ac86a0","Type":"ContainerStarted","Data":"fb4e7cd896b2f50d531fe82924ebfbe19b6aa45cac14c0d92e2cc2ea1334a713"} Oct 30 12:32:21.852504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.852469 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" event={"ID":"fdd40382-d780-4cab-941f-b2ef96ac86a0","Type":"ContainerStarted","Data":"d45926f41cb448514425256f23ca46400cd395fcd267c417a53874d24c824de4"} Oct 30 12:32:21.857424 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.857397 4413 generic.go:356] "Generic (PLEG): container finished" podID="2cb74db0-df6e-471c-bce4-7de34b2a33e3" containerID="55f66ad0552618147760f065b20735848b774f655c64da2e36479e18669a6de3" exitCode=0 Oct 30 12:32:21.858243 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.858227 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" Oct 30 12:32:21.866041 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.858400 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" event={"ID":"2cb74db0-df6e-471c-bce4-7de34b2a33e3","Type":"ContainerDied","Data":"55f66ad0552618147760f065b20735848b774f655c64da2e36479e18669a6de3"} Oct 30 12:32:21.866041 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.858446 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw" event={"ID":"2cb74db0-df6e-471c-bce4-7de34b2a33e3","Type":"ContainerDied","Data":"37c50fd8465ff9bb82c8c6dfbabbaaed6e418e325ec27498c51482a297af4e24"} Oct 30 12:32:21.866041 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.858473 4413 scope.go:122] "RemoveContainer" containerID="55f66ad0552618147760f065b20735848b774f655c64da2e36479e18669a6de3" Oct 30 12:32:21.871371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.869581 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:21.877174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.876219 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-workload-certs\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.877174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.876332 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-envoy\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.877174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.876365 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-workload-socket\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.877174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.876390 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-credential-socket\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.877174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.876418 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-data\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.877174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.876490 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkxhj\" (UniqueName: \"kubernetes.io/projected/c04e3112-1c05-40db-a2ff-659b04c79ffa-kube-api-access-dkxhj\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.877174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.876524 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04e3112-1c05-40db-a2ff-659b04c79ffa-istiod-ca-cert\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.877174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.876640 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-token\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.877174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.876664 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-workload-certs\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.877174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.876696 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-podinfo\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.877174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.876910 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-data\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.877806 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.877479 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-workload-socket\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.878559 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.878160 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04e3112-1c05-40db-a2ff-659b04c79ffa-istiod-ca-cert\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.878559 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.878433 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-credential-socket\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.878808 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.878626 4413 scope.go:122] "RemoveContainer" containerID="9653960bc2c2c48634dec6cb66996a6b53430357553c14e919ec81dbf4a4d690" Oct 30 12:32:21.880133 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.879771 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-envoy\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.880133 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.880079 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-podinfo\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.938551 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.932620 4413 scope.go:122] "RemoveContainer" containerID="55f66ad0552618147760f065b20735848b774f655c64da2e36479e18669a6de3" Oct 30 12:32:21.943811 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.942344 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-token\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.944386 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:32:21.944341 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55f66ad0552618147760f065b20735848b774f655c64da2e36479e18669a6de3\": container with ID starting with 55f66ad0552618147760f065b20735848b774f655c64da2e36479e18669a6de3 not found: ID does not exist" containerID="55f66ad0552618147760f065b20735848b774f655c64da2e36479e18669a6de3" Oct 30 12:32:21.944530 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.944393 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f66ad0552618147760f065b20735848b774f655c64da2e36479e18669a6de3"} err="failed to get container status \"55f66ad0552618147760f065b20735848b774f655c64da2e36479e18669a6de3\": rpc error: code = NotFound desc = could not find container \"55f66ad0552618147760f065b20735848b774f655c64da2e36479e18669a6de3\": container with ID starting with 55f66ad0552618147760f065b20735848b774f655c64da2e36479e18669a6de3 not found: ID does not exist" Oct 30 12:32:21.944530 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.944421 4413 scope.go:122] "RemoveContainer" containerID="9653960bc2c2c48634dec6cb66996a6b53430357553c14e919ec81dbf4a4d690" Oct 30 12:32:21.944530 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.944488 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:21.946032 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:32:21.945999 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9653960bc2c2c48634dec6cb66996a6b53430357553c14e919ec81dbf4a4d690\": container with ID starting with 9653960bc2c2c48634dec6cb66996a6b53430357553c14e919ec81dbf4a4d690 not found: ID does not exist" containerID="9653960bc2c2c48634dec6cb66996a6b53430357553c14e919ec81dbf4a4d690" Oct 30 12:32:21.946151 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.946035 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9653960bc2c2c48634dec6cb66996a6b53430357553c14e919ec81dbf4a4d690"} err="failed to get container status \"9653960bc2c2c48634dec6cb66996a6b53430357553c14e919ec81dbf4a4d690\": rpc error: code = NotFound desc = could not find container \"9653960bc2c2c48634dec6cb66996a6b53430357553c14e919ec81dbf4a4d690\": container with ID starting with 9653960bc2c2c48634dec6cb66996a6b53430357553c14e919ec81dbf4a4d690 not found: ID does not exist" Oct 30 12:32:21.959271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.947633 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" podStartSLOduration=2.947619471 podStartE2EDuration="2.947619471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:21.898889182 +0000 UTC m=+2295.008732058" watchObservedRunningTime="2026-10-30 12:32:21.947619471 +0000 UTC m=+2295.057462351" Oct 30 12:32:21.965004 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.961201 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkxhj\" (UniqueName: \"kubernetes.io/projected/c04e3112-1c05-40db-a2ff-659b04c79ffa-kube-api-access-dkxhj\") pod \"e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:21.985639 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.985518 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw"] Oct 30 12:32:21.988110 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:21.987957 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5c57af-openshift-default-7c49498d95-l9hlw"] Oct 30 12:32:22.054215 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.054177 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:22.059112 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.059082 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:22.274761 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:32:22.274720 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b50ec75_3d82_436e_914c_0d667dbeb3ae.slice/crio-824dd57dab0d0ccf82383cdd35a1e38f667949d606fc4278f4cd58388c36fe5c WatchSource:0}: Error finding container 824dd57dab0d0ccf82383cdd35a1e38f667949d606fc4278f4cd58388c36fe5c: Status 404 returned error can't find the container with id 824dd57dab0d0ccf82383cdd35a1e38f667949d606fc4278f4cd58388c36fe5c Oct 30 12:32:22.275194 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.275172 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn"] Oct 30 12:32:22.279328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.279293 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:32:22.279523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.279499 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:32:22.279668 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.279649 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:32:22.455363 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.455328 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw"] Oct 30 12:32:22.456043 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:32:22.455971 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04e3112_1c05_40db_a2ff_659b04c79ffa.slice/crio-7ce3fda7f8084ddf95b5845700da547ec57a95e37253c7d39347696d551d0321 WatchSource:0}: Error finding container 7ce3fda7f8084ddf95b5845700da547ec57a95e37253c7d39347696d551d0321: Status 404 returned error can't find the container with id 7ce3fda7f8084ddf95b5845700da547ec57a95e37253c7d39347696d551d0321 Oct 30 12:32:22.456549 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.456522 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:22.459476 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.459440 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:32:22.460191 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.459540 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:32:22.460347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.460317 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:32:22.871131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.871107 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 12:32:22.871352 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.871329 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" podUID="fdd40382-d780-4cab-941f-b2ef96ac86a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 12:32:22.878039 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.877715 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" event={"ID":"c04e3112-1c05-40db-a2ff-659b04c79ffa","Type":"ContainerStarted","Data":"67fb2fca7d5938841cde9d6d778cef4d236853fe9d5b72144e811a51046bbe89"} Oct 30 12:32:22.878039 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.877783 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" event={"ID":"c04e3112-1c05-40db-a2ff-659b04c79ffa","Type":"ContainerStarted","Data":"7ce3fda7f8084ddf95b5845700da547ec57a95e37253c7d39347696d551d0321"} Oct 30 12:32:22.880097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.880018 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" event={"ID":"8b50ec75-3d82-436e-914c-0d667dbeb3ae","Type":"ContainerStarted","Data":"80e9de8e7a06fcd241091381975114793df6b14db92239ae0ce38819dd78cf7e"} Oct 30 12:32:22.880097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.880064 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" event={"ID":"8b50ec75-3d82-436e-914c-0d667dbeb3ae","Type":"ContainerStarted","Data":"824dd57dab0d0ccf82383cdd35a1e38f667949d606fc4278f4cd58388c36fe5c"} Oct 30 12:32:22.956327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.938394 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" podStartSLOduration=1.938372777 podStartE2EDuration="1.938372777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:22.927608072 +0000 UTC m=+2296.037450957" watchObservedRunningTime="2026-10-30 12:32:22.938372777 +0000 UTC m=+2296.048215674" Oct 30 12:32:22.978275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:22.972983 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:23.079564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:23.077026 4413 patch_prober.go:29] interesting pod/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 12:32:23.079564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:23.077104 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" podUID="c04e3112-1c05-40db-a2ff-659b04c79ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 12:32:23.179274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:23.179218 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb74db0-df6e-471c-bce4-7de34b2a33e3" path="/var/lib/kubelet/pods/2cb74db0-df6e-471c-bce4-7de34b2a33e3/volumes" Oct 30 12:32:23.179704 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:23.179686 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:23.781528 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:23.781492 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:23.783242 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:23.783210 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 12:32:23.783360 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:23.783293 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" podUID="8b50ec75-3d82-436e-914c-0d667dbeb3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 12:32:23.876729 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:23.875539 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 12:32:23.876729 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:23.875600 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" podUID="fdd40382-d780-4cab-941f-b2ef96ac86a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 12:32:23.908275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:23.900425 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:24.049333 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.024427 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:24.083591 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.083543 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" podStartSLOduration=3.083527762 podStartE2EDuration="3.083527762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:22.990301228 +0000 UTC m=+2296.100144126" watchObservedRunningTime="2026-10-30 12:32:24.083527762 +0000 UTC m=+2297.193370651" Oct 30 12:32:24.083764 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.083719 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr"] Oct 30 12:32:24.088502 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.088480 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:24.088776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.088758 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr" Oct 30 12:32:24.126593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.126562 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-a124ab\"" Oct 30 12:32:24.201063 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.171580 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr"] Oct 30 12:32:24.213275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.212194 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-a124ab-dockercfg-rlwj9\"" Oct 30 12:32:24.335333 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.322448 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407bfb0f-7219-4558-bfa4-7c31fed7e3e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr\" (UID: \"407bfb0f-7219-4558-bfa4-7c31fed7e3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr" Oct 30 12:32:24.335333 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.322586 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6hd4\" (UniqueName: \"kubernetes.io/projected/407bfb0f-7219-4558-bfa4-7c31fed7e3e6-kube-api-access-g6hd4\") pod \"payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr\" (UID: \"407bfb0f-7219-4558-bfa4-7c31fed7e3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr" Oct 30 12:32:24.446309 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.432078 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6hd4\" (UniqueName: \"kubernetes.io/projected/407bfb0f-7219-4558-bfa4-7c31fed7e3e6-kube-api-access-g6hd4\") pod \"payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr\" (UID: \"407bfb0f-7219-4558-bfa4-7c31fed7e3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr" Oct 30 12:32:24.446309 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.432305 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407bfb0f-7219-4558-bfa4-7c31fed7e3e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr\" (UID: \"407bfb0f-7219-4558-bfa4-7c31fed7e3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr" Oct 30 12:32:24.446309 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.433476 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407bfb0f-7219-4558-bfa4-7c31fed7e3e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr\" (UID: \"407bfb0f-7219-4558-bfa4-7c31fed7e3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr" Oct 30 12:32:24.488935 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.488903 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6hd4\" (UniqueName: \"kubernetes.io/projected/407bfb0f-7219-4558-bfa4-7c31fed7e3e6-kube-api-access-g6hd4\") pod \"payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr\" (UID: \"407bfb0f-7219-4558-bfa4-7c31fed7e3e6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr" Oct 30 12:32:24.542727 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.542520 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr" Oct 30 12:32:24.782697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.782308 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 12:32:24.782697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.782367 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" podUID="8b50ec75-3d82-436e-914c-0d667dbeb3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 12:32:24.870790 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.870361 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 12:32:24.870790 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.870416 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" podUID="fdd40382-d780-4cab-941f-b2ef96ac86a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 12:32:24.948531 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.947610 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr"] Oct 30 12:32:24.955554 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.955450 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:24.956645 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:24.956447 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:32:25.781874 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:25.781843 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 12:32:25.782021 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:25.781896 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" podUID="8b50ec75-3d82-436e-914c-0d667dbeb3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 12:32:25.870199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:25.870156 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 12:32:25.870674 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:25.870214 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" podUID="fdd40382-d780-4cab-941f-b2ef96ac86a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 12:32:25.885494 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:25.885469 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:25.963385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:25.963355 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr" event={"ID":"407bfb0f-7219-4558-bfa4-7c31fed7e3e6","Type":"ContainerStarted","Data":"04c6ed8bfc4803b4e28ed298fddabac7632ff25a87c9795d0d056428877a61aa"} Oct 30 12:32:25.963552 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:25.963410 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr" event={"ID":"407bfb0f-7219-4558-bfa4-7c31fed7e3e6","Type":"ContainerStarted","Data":"69e3f6c99e60346e4ef80d33e62158368024e34f4c294e9c8eb580f035f2f44d"} Oct 30 12:32:25.963636 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:25.963551 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr" Oct 30 12:32:26.044331 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:26.036760 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:26.160321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:26.160273 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:26.286539 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:26.283904 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:26.783277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:26.782001 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 12:32:26.783277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:26.782047 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" podUID="8b50ec75-3d82-436e-914c-0d667dbeb3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 12:32:26.870713 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:26.870685 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 12:32:26.871084 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:26.870742 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" podUID="fdd40382-d780-4cab-941f-b2ef96ac86a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 12:32:27.782864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:27.782826 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 12:32:27.783062 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:27.782895 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" podUID="8b50ec75-3d82-436e-914c-0d667dbeb3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 12:32:27.871805 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:27.871348 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 12:32:27.871805 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:27.871414 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" podUID="fdd40382-d780-4cab-941f-b2ef96ac86a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 12:32:28.782109 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:28.782064 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 12:32:28.782397 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:28.782134 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" podUID="8b50ec75-3d82-436e-914c-0d667dbeb3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 12:32:28.870618 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:28.870583 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 12:32:28.870784 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:28.870651 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" podUID="fdd40382-d780-4cab-941f-b2ef96ac86a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 12:32:29.063447 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:29.063358 4413 patch_prober.go:29] interesting pod/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:32:29.063447 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:29.063416 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" podUID="2efc5e39-2e4f-4700-96d3-0b478cac830f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:29.540510 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:29.540454 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:29.657711 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:29.657664 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:29.782184 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:29.782150 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 12:32:29.782471 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:29.782213 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" podUID="8b50ec75-3d82-436e-914c-0d667dbeb3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 12:32:29.787627 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:29.787602 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:29.876460 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:29.876433 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:29.876955 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:29.876922 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:29.877761 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:29.877743 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:32:29.927812 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:29.927752 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr" podStartSLOduration=5.92773332 podStartE2EDuration="5.92773332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:26.046665362 +0000 UTC m=+2299.156508244" watchObservedRunningTime="2026-10-30 12:32:29.92773332 +0000 UTC m=+2303.037576203" Oct 30 12:32:29.930459 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:29.930429 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx"] Oct 30 12:32:29.936470 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:29.936429 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx" Oct 30 12:32:29.950967 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:29.950945 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-2461e8-dockercfg-xkclw\"" Oct 30 12:32:29.953231 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:29.953209 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-2461e8\"" Oct 30 12:32:29.956153 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:29.956124 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx"] Oct 30 12:32:30.191287 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:30.191174 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf96915-55fd-47b0-bace-c95cdb790cf2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx\" (UID: \"5cf96915-55fd-47b0-bace-c95cdb790cf2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx" Oct 30 12:32:30.191777 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:30.191326 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbp7v\" (UniqueName: \"kubernetes.io/projected/5cf96915-55fd-47b0-bace-c95cdb790cf2-kube-api-access-sbp7v\") pod \"payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx\" (UID: \"5cf96915-55fd-47b0-bace-c95cdb790cf2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx" Oct 30 12:32:30.293203 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:30.293165 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbp7v\" (UniqueName: \"kubernetes.io/projected/5cf96915-55fd-47b0-bace-c95cdb790cf2-kube-api-access-sbp7v\") pod \"payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx\" (UID: \"5cf96915-55fd-47b0-bace-c95cdb790cf2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx" Oct 30 12:32:30.294523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:30.293590 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf96915-55fd-47b0-bace-c95cdb790cf2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx\" (UID: \"5cf96915-55fd-47b0-bace-c95cdb790cf2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx" Oct 30 12:32:30.294523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:30.294326 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf96915-55fd-47b0-bace-c95cdb790cf2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx\" (UID: \"5cf96915-55fd-47b0-bace-c95cdb790cf2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx" Oct 30 12:32:30.316457 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:30.316422 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbp7v\" (UniqueName: \"kubernetes.io/projected/5cf96915-55fd-47b0-bace-c95cdb790cf2-kube-api-access-sbp7v\") pod \"payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx\" (UID: \"5cf96915-55fd-47b0-bace-c95cdb790cf2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx" Oct 30 12:32:30.556267 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:30.556218 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx" Oct 30 12:32:30.787294 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:30.787201 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:30.787719 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:30.787700 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:30.788323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:30.788307 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:32:30.864789 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:30.864765 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx"] Oct 30 12:32:30.865738 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:32:30.865717 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf96915_55fd_47b0_bace_c95cdb790cf2.slice/crio-81c78cc5525bf7e866cc8488165e6a5ad9ef0fea146c67a05df712cf09d386a0 WatchSource:0}: Error finding container 81c78cc5525bf7e866cc8488165e6a5ad9ef0fea146c67a05df712cf09d386a0: Status 404 returned error can't find the container with id 81c78cc5525bf7e866cc8488165e6a5ad9ef0fea146c67a05df712cf09d386a0 Oct 30 12:32:31.192036 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:31.192002 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx" event={"ID":"5cf96915-55fd-47b0-bace-c95cdb790cf2","Type":"ContainerStarted","Data":"99b470f3d937ffc6996ae4ea2ea8845c4d4afadb93a96720dac14240e551b1ec"} Oct 30 12:32:31.192523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:31.192063 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx" event={"ID":"5cf96915-55fd-47b0-bace-c95cdb790cf2","Type":"ContainerStarted","Data":"81c78cc5525bf7e866cc8488165e6a5ad9ef0fea146c67a05df712cf09d386a0"} Oct 30 12:32:31.192523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:31.192188 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx" Oct 30 12:32:31.194404 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:31.194377 4413 generic.go:356] "Generic (PLEG): container finished" podID="2efc5e39-2e4f-4700-96d3-0b478cac830f" containerID="db3e8b6e262d9fae95ea47f5f2210d5f9962b0bd0a67bff7ddd4140c529c6546" exitCode=0 Oct 30 12:32:31.194525 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:31.194446 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" event={"ID":"2efc5e39-2e4f-4700-96d3-0b478cac830f","Type":"ContainerDied","Data":"db3e8b6e262d9fae95ea47f5f2210d5f9962b0bd0a67bff7ddd4140c529c6546"} Oct 30 12:32:31.195161 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:31.195142 4413 scope.go:122] "RemoveContainer" containerID="db3e8b6e262d9fae95ea47f5f2210d5f9962b0bd0a67bff7ddd4140c529c6546" Oct 30 12:32:31.196640 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:31.196612 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:32:31.196721 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:31.196681 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:32:31.196721 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:31.196716 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:32:31.232198 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:31.232152 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx" podStartSLOduration=2.2321411700000002 podStartE2EDuration="2.23214117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:31.23068091 +0000 UTC m=+2304.340523810" watchObservedRunningTime="2026-10-30 12:32:31.23214117 +0000 UTC m=+2304.341984051" Oct 30 12:32:31.969263 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:31.969222 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:32.227874 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:32.227778 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" event={"ID":"2efc5e39-2e4f-4700-96d3-0b478cac830f","Type":"ContainerStarted","Data":"ff36dba9d4249848919f3a8857d19202cc4698a9be04c869e07ac61f114e5223"} Oct 30 12:32:32.766731 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:32.766698 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:32:32.767038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:32.766941 4413 patch_prober.go:29] interesting pod/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 12:32:32.767038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:32.767001 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" podUID="2efc5e39-2e4f-4700-96d3-0b478cac830f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 12:32:33.627785 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:33.627752 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:33.767781 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:33.767748 4413 patch_prober.go:29] interesting pod/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 12:32:33.768001 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:33.767805 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" podUID="2efc5e39-2e4f-4700-96d3-0b478cac830f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 12:32:34.767815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:34.767387 4413 patch_prober.go:29] interesting pod/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 12:32:34.767815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:34.767440 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" podUID="2efc5e39-2e4f-4700-96d3-0b478cac830f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 12:32:35.418276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:35.415600 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:35.784337 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:35.784090 4413 patch_prober.go:29] interesting pod/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 12:32:35.784337 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:35.784148 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" podUID="2efc5e39-2e4f-4700-96d3-0b478cac830f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 12:32:36.073729 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:32:36.073589 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:32:36.772273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:36.771501 4413 patch_prober.go:29] interesting pod/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 12:32:36.772273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:36.771563 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" podUID="2efc5e39-2e4f-4700-96d3-0b478cac830f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 12:32:36.894278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:36.886413 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" podUID="969651c2-7e9e-4b32-8e86-be72e29bd06b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:37.011783 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:37.010644 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr" Oct 30 12:32:37.122431 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:37.122337 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 12:32:37.153662 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:37.153624 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:37.172281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:37.166349 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:32:37.433194 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:37.433158 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:37.546012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:37.545976 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:37.767359 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:37.767274 4413 patch_prober.go:29] interesting pod/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 30 12:32:37.767359 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:37.767327 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" podUID="2efc5e39-2e4f-4700-96d3-0b478cac830f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 30 12:32:38.771150 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:38.771116 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:32:38.771619 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:38.771484 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:32:38.772319 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:38.772300 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:32:39.436967 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:39.436938 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:41.180636 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.180606 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm"] Oct 30 12:32:41.183893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.183877 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" Oct 30 12:32:41.207887 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.207858 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-a124ab-serving-cert\"" Oct 30 12:32:41.211568 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.211541 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638d3046-09df-44a7-864c-2586fe38ec94-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm\" (UID: \"638d3046-09df-44a7-864c-2586fe38ec94\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" Oct 30 12:32:41.211667 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.211635 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638d3046-09df-44a7-864c-2586fe38ec94-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm\" (UID: \"638d3046-09df-44a7-864c-2586fe38ec94\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" Oct 30 12:32:41.211756 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.211739 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcdlf\" (UniqueName: \"kubernetes.io/projected/638d3046-09df-44a7-864c-2586fe38ec94-kube-api-access-hcdlf\") pod \"maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm\" (UID: \"638d3046-09df-44a7-864c-2586fe38ec94\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" Oct 30 12:32:41.211821 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.211808 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/638d3046-09df-44a7-864c-2586fe38ec94-metrics-certs\") pod \"maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm\" (UID: \"638d3046-09df-44a7-864c-2586fe38ec94\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" Oct 30 12:32:41.212214 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.212194 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm"] Oct 30 12:32:41.281224 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.281183 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:41.312653 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.312616 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638d3046-09df-44a7-864c-2586fe38ec94-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm\" (UID: \"638d3046-09df-44a7-864c-2586fe38ec94\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" Oct 30 12:32:41.312808 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.312685 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638d3046-09df-44a7-864c-2586fe38ec94-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm\" (UID: \"638d3046-09df-44a7-864c-2586fe38ec94\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" Oct 30 12:32:41.312808 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.312756 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcdlf\" (UniqueName: \"kubernetes.io/projected/638d3046-09df-44a7-864c-2586fe38ec94-kube-api-access-hcdlf\") pod \"maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm\" (UID: \"638d3046-09df-44a7-864c-2586fe38ec94\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" Oct 30 12:32:41.312915 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.312829 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/638d3046-09df-44a7-864c-2586fe38ec94-metrics-certs\") pod \"maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm\" (UID: \"638d3046-09df-44a7-864c-2586fe38ec94\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" Oct 30 12:32:41.313482 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.313410 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638d3046-09df-44a7-864c-2586fe38ec94-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm\" (UID: \"638d3046-09df-44a7-864c-2586fe38ec94\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" Oct 30 12:32:41.315371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.315351 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/638d3046-09df-44a7-864c-2586fe38ec94-metrics-certs\") pod \"maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm\" (UID: \"638d3046-09df-44a7-864c-2586fe38ec94\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" Oct 30 12:32:41.315478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.315394 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638d3046-09df-44a7-864c-2586fe38ec94-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm\" (UID: \"638d3046-09df-44a7-864c-2586fe38ec94\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" Oct 30 12:32:41.336134 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.336103 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcdlf\" (UniqueName: \"kubernetes.io/projected/638d3046-09df-44a7-864c-2586fe38ec94-kube-api-access-hcdlf\") pod \"maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm\" (UID: \"638d3046-09df-44a7-864c-2586fe38ec94\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" Oct 30 12:32:41.408051 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.407995 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:41.498566 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.498476 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" Oct 30 12:32:41.803558 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:41.803529 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm"] Oct 30 12:32:41.804669 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:32:41.804644 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638d3046_09df_44a7_864c_2586fe38ec94.slice/crio-45255fe0ceb66783670e379d3f83b307a1a9f38bb565c582988b221d0849fa2a WatchSource:0}: Error finding container 45255fe0ceb66783670e379d3f83b307a1a9f38bb565c582988b221d0849fa2a: Status 404 returned error can't find the container with id 45255fe0ceb66783670e379d3f83b307a1a9f38bb565c582988b221d0849fa2a Oct 30 12:32:42.232034 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:42.232016 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx" Oct 30 12:32:42.405213 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:42.405177 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" event={"ID":"638d3046-09df-44a7-864c-2586fe38ec94","Type":"ContainerStarted","Data":"feddb8d8a85eda7f73abbd29713fa97f537b8ff05847cc84c5557e6212a1e490"} Oct 30 12:32:42.405431 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:42.405238 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" event={"ID":"638d3046-09df-44a7-864c-2586fe38ec94","Type":"ContainerStarted","Data":"45255fe0ceb66783670e379d3f83b307a1a9f38bb565c582988b221d0849fa2a"} Oct 30 12:32:42.405431 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:42.405304 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" Oct 30 12:32:42.449431 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:42.449380 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" podStartSLOduration=1.037453644 podStartE2EDuration="1.449366311s" totalImagesPullingTime="411.912667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:42.447678188 +0000 UTC m=+2315.557521095" watchObservedRunningTime="2026-10-30 12:32:42.449366311 +0000 UTC m=+2315.559209191" Oct 30 12:32:43.253666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:43.253619 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:45.079078 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:45.079051 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:46.624156 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:46.624116 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9"] Oct 30 12:32:46.630614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:46.630579 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:32:46.653081 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:46.653050 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9"] Oct 30 12:32:46.655872 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:46.655846 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-2461e8-serving-cert\"" Oct 30 12:32:46.686409 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:46.686375 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12bc929-1c97-4b32-a01a-250013d307f6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9\" (UID: \"a12bc929-1c97-4b32-a01a-250013d307f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:32:46.686608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:46.686545 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpc4f\" (UniqueName: \"kubernetes.io/projected/a12bc929-1c97-4b32-a01a-250013d307f6-kube-api-access-dpc4f\") pod \"maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9\" (UID: \"a12bc929-1c97-4b32-a01a-250013d307f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:32:46.686608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:46.686579 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12bc929-1c97-4b32-a01a-250013d307f6-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9\" (UID: \"a12bc929-1c97-4b32-a01a-250013d307f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:32:46.686740 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:46.686694 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12bc929-1c97-4b32-a01a-250013d307f6-metrics-certs\") pod \"maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9\" (UID: \"a12bc929-1c97-4b32-a01a-250013d307f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:32:46.716368 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:46.716340 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:46.788390 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:46.788356 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12bc929-1c97-4b32-a01a-250013d307f6-metrics-certs\") pod \"maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9\" (UID: \"a12bc929-1c97-4b32-a01a-250013d307f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:32:46.788583 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:46.788469 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12bc929-1c97-4b32-a01a-250013d307f6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9\" (UID: \"a12bc929-1c97-4b32-a01a-250013d307f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:32:46.788583 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:46.788561 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpc4f\" (UniqueName: \"kubernetes.io/projected/a12bc929-1c97-4b32-a01a-250013d307f6-kube-api-access-dpc4f\") pod \"maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9\" (UID: \"a12bc929-1c97-4b32-a01a-250013d307f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:32:46.788698 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:46.788589 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12bc929-1c97-4b32-a01a-250013d307f6-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9\" (UID: \"a12bc929-1c97-4b32-a01a-250013d307f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:32:46.788698 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:32:46.788599 4413 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-serving-cert: secret "maas-api-e2e-praxis-b-2461e8-serving-cert" not found Oct 30 12:32:46.788698 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:32:46.788667 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a12bc929-1c97-4b32-a01a-250013d307f6-maas-api-tls podName:a12bc929-1c97-4b32-a01a-250013d307f6 nodeName:}" failed. No retries permitted until 2026-10-30 12:32:47.288646671 +0000 UTC m=+2320.398489544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a12bc929-1c97-4b32-a01a-250013d307f6-maas-api-tls") pod "maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" (UID: "a12bc929-1c97-4b32-a01a-250013d307f6") : secret "maas-api-e2e-praxis-b-2461e8-serving-cert" not found Oct 30 12:32:46.789216 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:46.789197 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12bc929-1c97-4b32-a01a-250013d307f6-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9\" (UID: \"a12bc929-1c97-4b32-a01a-250013d307f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:32:46.790963 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:46.790943 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12bc929-1c97-4b32-a01a-250013d307f6-metrics-certs\") pod \"maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9\" (UID: \"a12bc929-1c97-4b32-a01a-250013d307f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:32:46.815382 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:46.815351 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpc4f\" (UniqueName: \"kubernetes.io/projected/a12bc929-1c97-4b32-a01a-250013d307f6-kube-api-access-dpc4f\") pod \"maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9\" (UID: \"a12bc929-1c97-4b32-a01a-250013d307f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:32:46.837292 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:46.837236 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:46.952677 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:46.952573 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:47.295601 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:47.295518 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12bc929-1c97-4b32-a01a-250013d307f6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9\" (UID: \"a12bc929-1c97-4b32-a01a-250013d307f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:32:47.298241 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:47.298218 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12bc929-1c97-4b32-a01a-250013d307f6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9\" (UID: \"a12bc929-1c97-4b32-a01a-250013d307f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:32:47.549847 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:47.549758 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:32:47.856557 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:47.856526 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9"] Oct 30 12:32:47.857287 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:32:47.857227 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12bc929_1c97_4b32_a01a_250013d307f6.slice/crio-1d49a4de6726838a7991cea6fe41707a5517acdf213d86798e07e615acbcbeed WatchSource:0}: Error finding container 1d49a4de6726838a7991cea6fe41707a5517acdf213d86798e07e615acbcbeed: Status 404 returned error can't find the container with id 1d49a4de6726838a7991cea6fe41707a5517acdf213d86798e07e615acbcbeed Oct 30 12:32:48.417694 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:48.417670 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" Oct 30 12:32:48.448204 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:48.448168 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" event={"ID":"a12bc929-1c97-4b32-a01a-250013d307f6","Type":"ContainerStarted","Data":"f1abfa75545528a102262ced0232e643ff01d03fa10fbc22086442bfdb3546af"} Oct 30 12:32:48.448364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:48.448221 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" event={"ID":"a12bc929-1c97-4b32-a01a-250013d307f6","Type":"ContainerStarted","Data":"1d49a4de6726838a7991cea6fe41707a5517acdf213d86798e07e615acbcbeed"} Oct 30 12:32:48.448364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:48.448288 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:32:48.469391 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:48.469361 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:48.510307 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:48.510181 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" podStartSLOduration=2.125177839 podStartE2EDuration="2.510162235s" totalImagesPullingTime="384.984396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:48.505089434 +0000 UTC m=+2321.614932340" watchObservedRunningTime="2026-10-30 12:32:48.510162235 +0000 UTC m=+2321.620005117" Oct 30 12:32:49.592590 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:49.592548 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh"] Oct 30 12:32:49.732684 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:49.732652 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:49.843372 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:49.843276 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:50.064775 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.064745 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q"] Oct 30 12:32:50.068188 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.068169 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.091167 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.091132 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q"] Oct 30 12:32:50.138419 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.138386 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.138626 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.138443 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqhg7\" (UniqueName: \"kubernetes.io/projected/73313e21-a547-4a5d-b1bc-cd154a4b7143-kube-api-access-wqhg7\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.138626 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.138528 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.138626 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.138560 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.138626 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.138585 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.138626 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.138615 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.138878 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.138665 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.138878 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.138697 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73313e21-a547-4a5d-b1bc-cd154a4b7143-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.139128 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.139104 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.178906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.178870 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:50.247327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.245009 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.247327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.245064 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqhg7\" (UniqueName: \"kubernetes.io/projected/73313e21-a547-4a5d-b1bc-cd154a4b7143-kube-api-access-wqhg7\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.247327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.245104 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.247327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.245133 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.247327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.245158 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.247327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.245190 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.247327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.245238 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.247327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.245293 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73313e21-a547-4a5d-b1bc-cd154a4b7143-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.247327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.245359 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.259273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.255121 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-workload-certs\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.259273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.255503 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-credential-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.259273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.255831 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-workload-socket\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.259273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.256144 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-data\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.259273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.258381 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73313e21-a547-4a5d-b1bc-cd154a4b7143-istiod-ca-cert\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.265463 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.263821 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-envoy\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.267924 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.267891 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-podinfo\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.287290 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.282503 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqhg7\" (UniqueName: \"kubernetes.io/projected/73313e21-a547-4a5d-b1bc-cd154a4b7143-kube-api-access-wqhg7\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.291391 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.291362 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-token\") pod \"e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.382186 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.382157 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:50.700325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.700298 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q"] Oct 30 12:32:50.701645 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:32:50.701623 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73313e21_a547_4a5d_b1bc_cd154a4b7143.slice/crio-7eb1214d120fb8a5ffd895cbf722a8fb3f027cf4359bc8020e1c7a618211fa88 WatchSource:0}: Error finding container 7eb1214d120fb8a5ffd895cbf722a8fb3f027cf4359bc8020e1c7a618211fa88: Status 404 returned error can't find the container with id 7eb1214d120fb8a5ffd895cbf722a8fb3f027cf4359bc8020e1c7a618211fa88 Oct 30 12:32:50.704216 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.704183 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:32:50.704316 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.704299 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:32:50.704368 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:50.704337 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:32:51.470901 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:51.470865 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" event={"ID":"73313e21-a547-4a5d-b1bc-cd154a4b7143","Type":"ContainerStarted","Data":"c4716799fa34e59bf287bdf168509b2b4bddb6e566891d313197f228207a19e3"} Oct 30 12:32:51.471082 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:51.470929 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" event={"ID":"73313e21-a547-4a5d-b1bc-cd154a4b7143","Type":"ContainerStarted","Data":"7eb1214d120fb8a5ffd895cbf722a8fb3f027cf4359bc8020e1c7a618211fa88"} Oct 30 12:32:51.589971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:51.589915 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" podStartSLOduration=1.589902725 podStartE2EDuration="1.589902725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:51.587766237 +0000 UTC m=+2324.697609122" watchObservedRunningTime="2026-10-30 12:32:51.589902725 +0000 UTC m=+2324.699745659" Oct 30 12:32:52.085136 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:52.085107 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:52.382790 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:52.382757 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:52.384079 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:52.384054 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 12:32:52.384180 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:52.384109 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" podUID="73313e21-a547-4a5d-b1bc-cd154a4b7143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 12:32:53.383240 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:53.383209 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 12:32:53.383655 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:53.383288 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" podUID="73313e21-a547-4a5d-b1bc-cd154a4b7143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 12:32:53.985137 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:53.985093 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:54.383289 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:54.383236 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 12:32:54.383732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:54.383321 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" podUID="73313e21-a547-4a5d-b1bc-cd154a4b7143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 12:32:54.459674 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:54.459649 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:32:54.615275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:54.613327 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" podUID="9d96d03b-bb3c-4841-a9e5-94996131da71" containerName="payload-processing" containerID="cri-o://9a560c59841c98fe45984d5fc7533afd1c2ce741ea772eeefb5f41bedefe8d98" gracePeriod=25 Oct 30 12:32:55.382860 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:55.382826 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 12:32:55.383063 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:55.382904 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" podUID="73313e21-a547-4a5d-b1bc-cd154a4b7143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 12:32:55.822810 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:55.822720 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:56.383266 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:56.383212 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 12:32:56.383446 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:56.383298 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" podUID="73313e21-a547-4a5d-b1bc-cd154a4b7143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 12:32:57.382981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:57.382941 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 12:32:57.383429 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:57.383006 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" podUID="73313e21-a547-4a5d-b1bc-cd154a4b7143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 12:32:57.591042 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:57.590997 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:58.335089 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:58.335052 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:32:58.386819 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:58.386794 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:58.387287 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:58.387243 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:58.387877 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:58.387861 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:32:59.367142 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:32:59.367113 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:00.858276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:00.857739 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" podUID="9d96d03b-bb3c-4841-a9e5-94996131da71" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:33:01.497831 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:01.497792 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:01.984276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:01.983367 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" podUID="969651c2-7e9e-4b32-8e86-be72e29bd06b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:02.158483 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:02.158433 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:02.181412 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:02.181368 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:02.664501 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:02.664464 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:03.554579 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:03.554545 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:03.917304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:03.917231 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:04.038020 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:04.037978 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:04.386739 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:04.386706 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:04.572271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:04.567729 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:05.346847 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:05.346792 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:05.861223 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:33:05.861185 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:33:07.250341 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:07.250309 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:07.661846 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:07.661798 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:07.780965 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:07.780828 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:07.826889 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:07.826862 4413 scope.go:122] "RemoveContainer" containerID="d7763c84e920fbf347e2233c04eceda565bc2fd883a184788f568ead10df20d5" Oct 30 12:33:08.958257 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:08.958216 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:09.386338 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:09.386235 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:09.508599 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:09.508561 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:10.745052 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:10.745016 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:10.810575 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:10.810538 4413 generic.go:356] "Generic (PLEG): container finished" podID="db669f7d-1af0-4f6d-9a65-f3295751ceb8" containerID="9c5df2ade0ea8f11cc2fbb19bdc805d5bff57ab62982e9957090a760284dc47f" exitCode=0 Oct 30 12:33:10.810721 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:10.810617 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" event={"ID":"db669f7d-1af0-4f6d-9a65-f3295751ceb8","Type":"ContainerDied","Data":"9c5df2ade0ea8f11cc2fbb19bdc805d5bff57ab62982e9957090a760284dc47f"} Oct 30 12:33:10.811389 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:10.811370 4413 scope.go:122] "RemoveContainer" containerID="9c5df2ade0ea8f11cc2fbb19bdc805d5bff57ab62982e9957090a760284dc47f" Oct 30 12:33:10.813157 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:10.813125 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:33:10.813275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:10.813198 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:33:10.813275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:10.813230 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:33:10.849415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:10.849377 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" podUID="9d96d03b-bb3c-4841-a9e5-94996131da71" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 12:33:10.998461 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:10.998358 4413 patch_prober.go:29] interesting pod/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:33:10.998461 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:10.998416 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" podUID="c04e3112-1c05-40db-a2ff-659b04c79ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:11.286507 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:11.286468 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:11.822921 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:11.822888 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" event={"ID":"db669f7d-1af0-4f6d-9a65-f3295751ceb8","Type":"ContainerStarted","Data":"8dd2439f66f5bed3dd7c75c697cca5d737550a2abb9e1e0813bd3fea66784228"} Oct 30 12:33:11.824528 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:11.824504 4413 generic.go:356] "Generic (PLEG): container finished" podID="c04e3112-1c05-40db-a2ff-659b04c79ffa" containerID="67fb2fca7d5938841cde9d6d778cef4d236853fe9d5b72144e811a51046bbe89" exitCode=0 Oct 30 12:33:11.824648 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:11.824577 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" event={"ID":"c04e3112-1c05-40db-a2ff-659b04c79ffa","Type":"ContainerDied","Data":"67fb2fca7d5938841cde9d6d778cef4d236853fe9d5b72144e811a51046bbe89"} Oct 30 12:33:11.824930 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:11.824917 4413 scope.go:122] "RemoveContainer" containerID="67fb2fca7d5938841cde9d6d778cef4d236853fe9d5b72144e811a51046bbe89" Oct 30 12:33:11.826558 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:11.826533 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:33:11.826638 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:11.826599 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:33:11.826638 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:11.826633 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:33:11.988335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:11.987436 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:33:11.988335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:11.987895 4413 patch_prober.go:29] interesting pod/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 12:33:11.988335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:11.987943 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" podUID="db669f7d-1af0-4f6d-9a65-f3295751ceb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 12:33:12.416274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:12.415794 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:12.846481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:12.846393 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" event={"ID":"c04e3112-1c05-40db-a2ff-659b04c79ffa","Type":"ContainerStarted","Data":"f8d31e66e3d671a2b3a6f0cc0e858d9bef4fc8471ff47132e71c9e8c8de3d252"} Oct 30 12:33:13.012274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:13.008013 4413 patch_prober.go:29] interesting pod/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 12:33:13.012274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:13.008109 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" podUID="db669f7d-1af0-4f6d-9a65-f3295751ceb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 12:33:13.055068 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:13.055040 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:33:13.055068 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:13.055066 4413 patch_prober.go:29] interesting pod/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 12:33:13.055315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:13.055109 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" podUID="c04e3112-1c05-40db-a2ff-659b04c79ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 12:33:13.987884 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:13.987844 4413 patch_prober.go:29] interesting pod/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 12:33:13.988341 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:13.987934 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" podUID="db669f7d-1af0-4f6d-9a65-f3295751ceb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 12:33:14.054964 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:14.054931 4413 patch_prober.go:29] interesting pod/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 12:33:14.055159 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:14.054985 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" podUID="c04e3112-1c05-40db-a2ff-659b04c79ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 12:33:14.058503 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:14.058478 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:14.869650 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:14.869617 4413 generic.go:356] "Generic (PLEG): container finished" podID="9d96d03b-bb3c-4841-a9e5-94996131da71" containerID="9a560c59841c98fe45984d5fc7533afd1c2ce741ea772eeefb5f41bedefe8d98" exitCode=0 Oct 30 12:33:14.869969 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:14.869663 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" event={"ID":"9d96d03b-bb3c-4841-a9e5-94996131da71","Type":"ContainerDied","Data":"9a560c59841c98fe45984d5fc7533afd1c2ce741ea772eeefb5f41bedefe8d98"} Oct 30 12:33:14.987580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:14.987549 4413 patch_prober.go:29] interesting pod/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 12:33:14.987733 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:14.987595 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" podUID="db669f7d-1af0-4f6d-9a65-f3295751ceb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 12:33:15.054017 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:15.053788 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" Oct 30 12:33:15.057273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:15.056739 4413 patch_prober.go:29] interesting pod/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 12:33:15.057273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:15.056778 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" podUID="c04e3112-1c05-40db-a2ff-659b04c79ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 12:33:15.079219 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:15.079197 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d96d03b-bb3c-4841-a9e5-94996131da71-kube-api-access-hjc4r\" (UniqueName: \"kubernetes.io/projected/9d96d03b-bb3c-4841-a9e5-94996131da71-kube-api-access-hjc4r\") pod \"9d96d03b-bb3c-4841-a9e5-94996131da71\" (UID: \"9d96d03b-bb3c-4841-a9e5-94996131da71\") " Oct 30 12:33:15.079391 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:15.079310 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d96d03b-bb3c-4841-a9e5-94996131da71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d96d03b-bb3c-4841-a9e5-94996131da71-plugins-config-volume\") pod \"9d96d03b-bb3c-4841-a9e5-94996131da71\" (UID: \"9d96d03b-bb3c-4841-a9e5-94996131da71\") " Oct 30 12:33:15.080524 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:15.080498 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d96d03b-bb3c-4841-a9e5-94996131da71-plugins-config-volume" pod "9d96d03b-bb3c-4841-a9e5-94996131da71" (UID: "9d96d03b-bb3c-4841-a9e5-94996131da71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:15.081845 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:15.081817 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d96d03b-bb3c-4841-a9e5-94996131da71-kube-api-access-hjc4r" pod "9d96d03b-bb3c-4841-a9e5-94996131da71" (UID: "9d96d03b-bb3c-4841-a9e5-94996131da71"). InnerVolumeSpecName "kube-api-access-hjc4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:15.181808 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:15.181775 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d96d03b-bb3c-4841-a9e5-94996131da71-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:15.181808 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:15.181801 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjc4r\" (UniqueName: \"kubernetes.io/projected/9d96d03b-bb3c-4841-a9e5-94996131da71-kube-api-access-hjc4r\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:15.650280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:15.634035 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:15.881105 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:15.881072 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" event={"ID":"9d96d03b-bb3c-4841-a9e5-94996131da71","Type":"ContainerDied","Data":"efd0af80af247da99307999e9decd669ed822f6e362766e3ec43e053cc9c3206"} Oct 30 12:33:15.881295 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:15.881082 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh" Oct 30 12:33:15.881295 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:15.881168 4413 scope.go:122] "RemoveContainer" containerID="9a560c59841c98fe45984d5fc7533afd1c2ce741ea772eeefb5f41bedefe8d98" Oct 30 12:33:15.985857 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:15.985825 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh"] Oct 30 12:33:15.988196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:15.988171 4413 patch_prober.go:29] interesting pod/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 12:33:15.988504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:15.988476 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" podUID="db669f7d-1af0-4f6d-9a65-f3295751ceb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 12:33:16.011934 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:16.011895 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c44eee-67d86f69bd-hvbnh"] Oct 30 12:33:16.056698 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:16.056665 4413 patch_prober.go:29] interesting pod/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 12:33:16.058481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:16.058446 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" podUID="c04e3112-1c05-40db-a2ff-659b04c79ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 12:33:16.320663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:16.320539 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:16.437400 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:16.434815 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:16.550212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:16.550180 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:16.973716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:16.973672 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:16.987652 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:16.987622 4413 patch_prober.go:29] interesting pod/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 12:33:16.987797 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:16.987673 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" podUID="db669f7d-1af0-4f6d-9a65-f3295751ceb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 12:33:17.031549 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:17.031520 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d96d03b-bb3c-4841-a9e5-94996131da71" path="/var/lib/kubelet/pods/9d96d03b-bb3c-4841-a9e5-94996131da71/volumes" Oct 30 12:33:17.055089 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:17.055062 4413 patch_prober.go:29] interesting pod/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 12:33:17.055236 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:17.055113 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" podUID="c04e3112-1c05-40db-a2ff-659b04c79ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 12:33:17.993109 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:17.993080 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:33:17.993654 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:17.993530 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:33:17.994550 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:17.994519 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:33:18.055842 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:18.055806 4413 patch_prober.go:29] interesting pod/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 12:33:18.056025 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:18.055869 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" podUID="c04e3112-1c05-40db-a2ff-659b04c79ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 12:33:18.287984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:18.287903 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:18.680614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:18.680582 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:18.810997 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:18.810961 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:19.059576 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:19.059495 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:33:19.059961 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:19.059864 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:33:19.060489 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:19.060471 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:33:20.163183 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:20.163149 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:21.852354 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:21.852320 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:21.957413 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:21.957378 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h"] Oct 30 12:33:22.078497 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:22.078467 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:24.579072 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:24.579030 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:24.744894 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:24.744851 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:25.496607 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:25.496575 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm"] Oct 30 12:33:25.562350 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:25.562317 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x"] Oct 30 12:33:25.700934 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:25.700879 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:25.837487 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:25.837406 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:26.720562 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:26.720526 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:26.856295 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:26.856107 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:28.742038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:28.741998 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:30.451873 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:30.451840 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:30.497062 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:30.496987 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm" podUID="b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9" containerName="payload-processing" containerID="cri-o://e20e52cc3283bdf98c2b9dc0b7ac87794fc64e17da5fb71608461fec2290d36d" gracePeriod=25 Oct 30 12:33:30.563423 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:30.563368 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x" podUID="95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6" containerName="payload-pre-processing" containerID="cri-o://9ed159526f223e57dc49588a7cb4976e8e84b0823cf4284e812106a53206d07b" gracePeriod=25 Oct 30 12:33:30.839703 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:30.839671 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm" Oct 30 12:33:30.874422 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:30.874398 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x" Oct 30 12:33:30.927188 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:30.927155 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6-plugins-config-volume\") pod \"95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6\" (UID: \"95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6\") " Oct 30 12:33:30.927188 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:30.927193 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6-kube-api-access-cb6qh\" (UniqueName: \"kubernetes.io/projected/95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6-kube-api-access-cb6qh\") pod \"95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6\" (UID: \"95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6\") " Oct 30 12:33:30.927498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:30.927318 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9-kube-api-access-m2v59\" (UniqueName: \"kubernetes.io/projected/b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9-kube-api-access-m2v59\") pod \"b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9\" (UID: \"b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9\") " Oct 30 12:33:30.927498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:30.927398 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9-plugins-config-volume\") pod \"b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9\" (UID: \"b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9\") " Oct 30 12:33:30.927651 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:30.927623 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6-plugins-config-volume" pod "95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6" (UID: "95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:30.927833 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:30.927799 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9-plugins-config-volume" pod "b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9" (UID: "b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:30.928364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:30.928341 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:30.928448 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:30.928370 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:30.929921 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:30.929892 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6-kube-api-access-cb6qh" pod "95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6" (UID: "95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6"). InnerVolumeSpecName "kube-api-access-cb6qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:30.929921 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:30.929903 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9-kube-api-access-m2v59" pod "b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9" (UID: "b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9"). InnerVolumeSpecName "kube-api-access-m2v59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:31.029421 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.029354 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb6qh\" (UniqueName: \"kubernetes.io/projected/95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6-kube-api-access-cb6qh\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:31.029421 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.029378 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2v59\" (UniqueName: \"kubernetes.io/projected/b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9-kube-api-access-m2v59\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:31.259682 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.259646 4413 generic.go:356] "Generic (PLEG): container finished" podID="b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9" containerID="e20e52cc3283bdf98c2b9dc0b7ac87794fc64e17da5fb71608461fec2290d36d" exitCode=0 Oct 30 12:33:31.259868 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.259697 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm" Oct 30 12:33:31.259868 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.259726 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm" event={"ID":"b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9","Type":"ContainerDied","Data":"e20e52cc3283bdf98c2b9dc0b7ac87794fc64e17da5fb71608461fec2290d36d"} Oct 30 12:33:31.259868 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.259773 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm" event={"ID":"b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9","Type":"ContainerDied","Data":"c59a9661582e2558b8247947b980f050e8218caec913590e26f6f308e3a34ffa"} Oct 30 12:33:31.259868 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.259801 4413 scope.go:122] "RemoveContainer" containerID="e20e52cc3283bdf98c2b9dc0b7ac87794fc64e17da5fb71608461fec2290d36d" Oct 30 12:33:31.261097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.261071 4413 generic.go:356] "Generic (PLEG): container finished" podID="95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6" containerID="9ed159526f223e57dc49588a7cb4976e8e84b0823cf4284e812106a53206d07b" exitCode=0 Oct 30 12:33:31.261197 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.261117 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x" Oct 30 12:33:31.261197 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.261157 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x" event={"ID":"95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6","Type":"ContainerDied","Data":"9ed159526f223e57dc49588a7cb4976e8e84b0823cf4284e812106a53206d07b"} Oct 30 12:33:31.261349 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.261203 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x" event={"ID":"95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6","Type":"ContainerDied","Data":"de2b1fff7637f2a373e90cdc629c95fd071d13e1d1c08b1b0130911b3ab6397a"} Oct 30 12:33:31.269604 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.269587 4413 scope.go:122] "RemoveContainer" containerID="e20e52cc3283bdf98c2b9dc0b7ac87794fc64e17da5fb71608461fec2290d36d" Oct 30 12:33:31.269908 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:33:31.269884 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20e52cc3283bdf98c2b9dc0b7ac87794fc64e17da5fb71608461fec2290d36d\": container with ID starting with e20e52cc3283bdf98c2b9dc0b7ac87794fc64e17da5fb71608461fec2290d36d not found: ID does not exist" containerID="e20e52cc3283bdf98c2b9dc0b7ac87794fc64e17da5fb71608461fec2290d36d" Oct 30 12:33:31.270006 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.269917 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20e52cc3283bdf98c2b9dc0b7ac87794fc64e17da5fb71608461fec2290d36d"} err="failed to get container status \"e20e52cc3283bdf98c2b9dc0b7ac87794fc64e17da5fb71608461fec2290d36d\": rpc error: code = NotFound desc = could not find container \"e20e52cc3283bdf98c2b9dc0b7ac87794fc64e17da5fb71608461fec2290d36d\": container with ID starting with e20e52cc3283bdf98c2b9dc0b7ac87794fc64e17da5fb71608461fec2290d36d not found: ID does not exist" Oct 30 12:33:31.270006 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.269942 4413 scope.go:122] "RemoveContainer" containerID="9ed159526f223e57dc49588a7cb4976e8e84b0823cf4284e812106a53206d07b" Oct 30 12:33:31.280381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.280363 4413 scope.go:122] "RemoveContainer" containerID="9ed159526f223e57dc49588a7cb4976e8e84b0823cf4284e812106a53206d07b" Oct 30 12:33:31.280674 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:33:31.280654 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed159526f223e57dc49588a7cb4976e8e84b0823cf4284e812106a53206d07b\": container with ID starting with 9ed159526f223e57dc49588a7cb4976e8e84b0823cf4284e812106a53206d07b not found: ID does not exist" containerID="9ed159526f223e57dc49588a7cb4976e8e84b0823cf4284e812106a53206d07b" Oct 30 12:33:31.280726 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.280683 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed159526f223e57dc49588a7cb4976e8e84b0823cf4284e812106a53206d07b"} err="failed to get container status \"9ed159526f223e57dc49588a7cb4976e8e84b0823cf4284e812106a53206d07b\": rpc error: code = NotFound desc = could not find container \"9ed159526f223e57dc49588a7cb4976e8e84b0823cf4284e812106a53206d07b\": container with ID starting with 9ed159526f223e57dc49588a7cb4976e8e84b0823cf4284e812106a53206d07b not found: ID does not exist" Oct 30 12:33:31.310884 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.310859 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm"] Oct 30 12:33:31.314422 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.314401 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0172c6-795556c9b6-mkssm"] Oct 30 12:33:31.347622 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.347592 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x"] Oct 30 12:33:31.351144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:31.351124 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0172c6-84c4d8584c-44b9x"] Oct 30 12:33:32.020761 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:32.020732 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:33.022310 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:33.022279 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6" path="/var/lib/kubelet/pods/95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6/volumes" Oct 30 12:33:33.022712 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:33.022698 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9" path="/var/lib/kubelet/pods/b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9/volumes" Oct 30 12:33:33.622494 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:33.622440 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:34.984440 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:34.984409 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd"] Oct 30 12:33:34.984927 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:34.984604 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" podUID="64cfb9ee-656e-4d1b-af81-1a73742935b7" containerName="maas-api" containerID="cri-o://5f2ec7460944df933c968edf735d7d3fb9a89436f6857ec294d5e46ddfa5a27b" gracePeriod=30 Oct 30 12:33:35.049456 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.049419 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:35.301788 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.301759 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" Oct 30 12:33:35.316332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.315992 4413 generic.go:356] "Generic (PLEG): container finished" podID="64cfb9ee-656e-4d1b-af81-1a73742935b7" containerID="5f2ec7460944df933c968edf735d7d3fb9a89436f6857ec294d5e46ddfa5a27b" exitCode=0 Oct 30 12:33:35.316332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.316113 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" event={"ID":"64cfb9ee-656e-4d1b-af81-1a73742935b7","Type":"ContainerDied","Data":"5f2ec7460944df933c968edf735d7d3fb9a89436f6857ec294d5e46ddfa5a27b"} Oct 30 12:33:35.316332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.316195 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" event={"ID":"64cfb9ee-656e-4d1b-af81-1a73742935b7","Type":"ContainerDied","Data":"46f0613a36e5bb74f07a2fb8ef3e68e6935d3523213559a38e85cde7880f9672"} Oct 30 12:33:35.316332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.316235 4413 scope.go:122] "RemoveContainer" containerID="5f2ec7460944df933c968edf735d7d3fb9a89436f6857ec294d5e46ddfa5a27b" Oct 30 12:33:35.330658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.330636 4413 scope.go:122] "RemoveContainer" containerID="5f2ec7460944df933c968edf735d7d3fb9a89436f6857ec294d5e46ddfa5a27b" Oct 30 12:33:35.330935 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:33:35.330913 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f2ec7460944df933c968edf735d7d3fb9a89436f6857ec294d5e46ddfa5a27b\": container with ID starting with 5f2ec7460944df933c968edf735d7d3fb9a89436f6857ec294d5e46ddfa5a27b not found: ID does not exist" containerID="5f2ec7460944df933c968edf735d7d3fb9a89436f6857ec294d5e46ddfa5a27b" Oct 30 12:33:35.331007 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.330949 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f2ec7460944df933c968edf735d7d3fb9a89436f6857ec294d5e46ddfa5a27b"} err="failed to get container status \"5f2ec7460944df933c968edf735d7d3fb9a89436f6857ec294d5e46ddfa5a27b\": rpc error: code = NotFound desc = could not find container \"5f2ec7460944df933c968edf735d7d3fb9a89436f6857ec294d5e46ddfa5a27b\": container with ID starting with 5f2ec7460944df933c968edf735d7d3fb9a89436f6857ec294d5e46ddfa5a27b not found: ID does not exist" Oct 30 12:33:35.387504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.387466 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64cfb9ee-656e-4d1b-af81-1a73742935b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64cfb9ee-656e-4d1b-af81-1a73742935b7-maas-api-tls\") pod \"64cfb9ee-656e-4d1b-af81-1a73742935b7\" (UID: \"64cfb9ee-656e-4d1b-af81-1a73742935b7\") " Oct 30 12:33:35.387770 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.387694 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64cfb9ee-656e-4d1b-af81-1a73742935b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/64cfb9ee-656e-4d1b-af81-1a73742935b7-metrics-certs\") pod \"64cfb9ee-656e-4d1b-af81-1a73742935b7\" (UID: \"64cfb9ee-656e-4d1b-af81-1a73742935b7\") " Oct 30 12:33:35.387770 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.387745 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64cfb9ee-656e-4d1b-af81-1a73742935b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64cfb9ee-656e-4d1b-af81-1a73742935b7-openshift-service-ca\") pod \"64cfb9ee-656e-4d1b-af81-1a73742935b7\" (UID: \"64cfb9ee-656e-4d1b-af81-1a73742935b7\") " Oct 30 12:33:35.387903 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.387785 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64cfb9ee-656e-4d1b-af81-1a73742935b7-kube-api-access-s58qv\" (UniqueName: \"kubernetes.io/projected/64cfb9ee-656e-4d1b-af81-1a73742935b7-kube-api-access-s58qv\") pod \"64cfb9ee-656e-4d1b-af81-1a73742935b7\" (UID: \"64cfb9ee-656e-4d1b-af81-1a73742935b7\") " Oct 30 12:33:35.388907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.388846 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64cfb9ee-656e-4d1b-af81-1a73742935b7-openshift-service-ca" pod "64cfb9ee-656e-4d1b-af81-1a73742935b7" (UID: "64cfb9ee-656e-4d1b-af81-1a73742935b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:35.390290 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.390243 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64cfb9ee-656e-4d1b-af81-1a73742935b7-metrics-certs" pod "64cfb9ee-656e-4d1b-af81-1a73742935b7" (UID: "64cfb9ee-656e-4d1b-af81-1a73742935b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:35.391381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.391120 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64cfb9ee-656e-4d1b-af81-1a73742935b7-maas-api-tls" pod "64cfb9ee-656e-4d1b-af81-1a73742935b7" (UID: "64cfb9ee-656e-4d1b-af81-1a73742935b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:35.391381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.391232 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64cfb9ee-656e-4d1b-af81-1a73742935b7-kube-api-access-s58qv" pod "64cfb9ee-656e-4d1b-af81-1a73742935b7" (UID: "64cfb9ee-656e-4d1b-af81-1a73742935b7"). InnerVolumeSpecName "kube-api-access-s58qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:35.493284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.492489 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64cfb9ee-656e-4d1b-af81-1a73742935b7-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:35.493284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.492527 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64cfb9ee-656e-4d1b-af81-1a73742935b7-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:35.493284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.492545 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64cfb9ee-656e-4d1b-af81-1a73742935b7-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:35.493284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.492559 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s58qv\" (UniqueName: \"kubernetes.io/projected/64cfb9ee-656e-4d1b-af81-1a73742935b7-kube-api-access-s58qv\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:35.538823 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.538730 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:35.697766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.697724 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:35.825296 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:33:35.825199 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:33:35.831743 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:35.831700 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:36.258729 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:36.258697 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:36.322996 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:36.322966 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd" Oct 30 12:33:36.373867 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:36.373827 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd"] Oct 30 12:33:36.385651 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:36.385624 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0172c6-84bdb5ff7d-q57gd"] Oct 30 12:33:36.988201 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:36.988160 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" podUID="969651c2-7e9e-4b32-8e86-be72e29bd06b" containerName="main" containerID="cri-o://465bf9d43552f6e6a55098c0d1f304089e33fc3bee449dcdaeed3bde13e5c767" gracePeriod=45 Oct 30 12:33:37.021607 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.021582 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64cfb9ee-656e-4d1b-af81-1a73742935b7" path="/var/lib/kubelet/pods/64cfb9ee-656e-4d1b-af81-1a73742935b7/volumes" Oct 30 12:33:37.265405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.265381 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:33:37.315903 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.315873 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-model-cache\") pod \"969651c2-7e9e-4b32-8e86-be72e29bd06b\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " Oct 30 12:33:37.316036 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.315977 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-dshm\") pod \"969651c2-7e9e-4b32-8e86-be72e29bd06b\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " Oct 30 12:33:37.316078 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.316044 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-tmp-dir\") pod \"969651c2-7e9e-4b32-8e86-be72e29bd06b\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " Oct 30 12:33:37.316146 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.316130 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/969651c2-7e9e-4b32-8e86-be72e29bd06b-tls-certs\" (UniqueName: \"kubernetes.io/secret/969651c2-7e9e-4b32-8e86-be72e29bd06b-tls-certs\") pod \"969651c2-7e9e-4b32-8e86-be72e29bd06b\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " Oct 30 12:33:37.316191 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.316163 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-home\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-home\") pod \"969651c2-7e9e-4b32-8e86-be72e29bd06b\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " Oct 30 12:33:37.316225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.316213 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/969651c2-7e9e-4b32-8e86-be72e29bd06b-kube-api-access-85ck6\" (UniqueName: \"kubernetes.io/projected/969651c2-7e9e-4b32-8e86-be72e29bd06b-kube-api-access-85ck6\") pod \"969651c2-7e9e-4b32-8e86-be72e29bd06b\" (UID: \"969651c2-7e9e-4b32-8e86-be72e29bd06b\") " Oct 30 12:33:37.317537 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.317393 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-tmp-dir" pod "969651c2-7e9e-4b32-8e86-be72e29bd06b" (UID: "969651c2-7e9e-4b32-8e86-be72e29bd06b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:33:37.317537 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.317433 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-home" pod "969651c2-7e9e-4b32-8e86-be72e29bd06b" (UID: "969651c2-7e9e-4b32-8e86-be72e29bd06b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:33:37.317833 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.317808 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-model-cache" pod "969651c2-7e9e-4b32-8e86-be72e29bd06b" (UID: "969651c2-7e9e-4b32-8e86-be72e29bd06b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:33:37.319394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.319370 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969651c2-7e9e-4b32-8e86-be72e29bd06b-kube-api-access-85ck6" pod "969651c2-7e9e-4b32-8e86-be72e29bd06b" (UID: "969651c2-7e9e-4b32-8e86-be72e29bd06b"). InnerVolumeSpecName "kube-api-access-85ck6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:37.320142 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.320114 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/969651c2-7e9e-4b32-8e86-be72e29bd06b-tls-certs" pod "969651c2-7e9e-4b32-8e86-be72e29bd06b" (UID: "969651c2-7e9e-4b32-8e86-be72e29bd06b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:33:37.320370 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.320347 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-dshm" pod "969651c2-7e9e-4b32-8e86-be72e29bd06b" (UID: "969651c2-7e9e-4b32-8e86-be72e29bd06b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:33:37.333963 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.333937 4413 generic.go:356] "Generic (PLEG): container finished" podID="969651c2-7e9e-4b32-8e86-be72e29bd06b" containerID="465bf9d43552f6e6a55098c0d1f304089e33fc3bee449dcdaeed3bde13e5c767" exitCode=0 Oct 30 12:33:37.334087 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.333988 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" Oct 30 12:33:37.334087 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.334027 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" event={"ID":"969651c2-7e9e-4b32-8e86-be72e29bd06b","Type":"ContainerDied","Data":"465bf9d43552f6e6a55098c0d1f304089e33fc3bee449dcdaeed3bde13e5c767"} Oct 30 12:33:37.334087 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.334081 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h" event={"ID":"969651c2-7e9e-4b32-8e86-be72e29bd06b","Type":"ContainerDied","Data":"de5bc641a7c2e2db055082139bcbfce1e5287d96ee80c2f16134baa655f7045d"} Oct 30 12:33:37.334294 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.334106 4413 scope.go:122] "RemoveContainer" containerID="465bf9d43552f6e6a55098c0d1f304089e33fc3bee449dcdaeed3bde13e5c767" Oct 30 12:33:37.356238 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.356217 4413 scope.go:122] "RemoveContainer" containerID="465bf9d43552f6e6a55098c0d1f304089e33fc3bee449dcdaeed3bde13e5c767" Oct 30 12:33:37.356575 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:33:37.356556 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"465bf9d43552f6e6a55098c0d1f304089e33fc3bee449dcdaeed3bde13e5c767\": container with ID starting with 465bf9d43552f6e6a55098c0d1f304089e33fc3bee449dcdaeed3bde13e5c767 not found: ID does not exist" containerID="465bf9d43552f6e6a55098c0d1f304089e33fc3bee449dcdaeed3bde13e5c767" Oct 30 12:33:37.356625 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.356587 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"465bf9d43552f6e6a55098c0d1f304089e33fc3bee449dcdaeed3bde13e5c767"} err="failed to get container status \"465bf9d43552f6e6a55098c0d1f304089e33fc3bee449dcdaeed3bde13e5c767\": rpc error: code = NotFound desc = could not find container \"465bf9d43552f6e6a55098c0d1f304089e33fc3bee449dcdaeed3bde13e5c767\": container with ID starting with 465bf9d43552f6e6a55098c0d1f304089e33fc3bee449dcdaeed3bde13e5c767 not found: ID does not exist" Oct 30 12:33:37.388876 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.388851 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h"] Oct 30 12:33:37.391727 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.391709 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-c44eee/praxis-route-c44eee-kserve-566795949c-hz74h"] Oct 30 12:33:37.418436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.418407 4413 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:37.418436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.418433 4413 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/969651c2-7e9e-4b32-8e86-be72e29bd06b-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:37.418436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.418441 4413 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:37.418649 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.418453 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85ck6\" (UniqueName: \"kubernetes.io/projected/969651c2-7e9e-4b32-8e86-be72e29bd06b-kube-api-access-85ck6\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:37.418649 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.418462 4413 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:37.418649 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.418471 4413 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/969651c2-7e9e-4b32-8e86-be72e29bd06b-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:37.552583 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:37.552494 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:39.022301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:39.022234 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="969651c2-7e9e-4b32-8e86-be72e29bd06b" path="/var/lib/kubelet/pods/969651c2-7e9e-4b32-8e86-be72e29bd06b/volumes" Oct 30 12:33:39.584275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:39.580364 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:41.118274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:41.117330 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:41.142273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:41.136377 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:41.237280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:41.236420 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:41.282278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:41.279521 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:33:42.963595 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:42.963561 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:43.098893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:43.098861 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:43.295782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:43.295691 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84"] Oct 30 12:33:43.297745 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:43.297674 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" podUID="2efc5e39-2e4f-4700-96d3-0b478cac830f" containerName="istio-proxy" containerID="cri-o://ff36dba9d4249848919f3a8857d19202cc4698a9be04c869e07ac61f114e5223" gracePeriod=30 Oct 30 12:33:43.342160 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:43.342129 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:43.479539 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:43.479510 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:45.790964 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:45.790927 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:46.579635 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:46.579597 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:47.554790 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:47.554755 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:48.501646 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.499266 4413 generic.go:356] "Generic (PLEG): container finished" podID="2efc5e39-2e4f-4700-96d3-0b478cac830f" containerID="ff36dba9d4249848919f3a8857d19202cc4698a9be04c869e07ac61f114e5223" exitCode=0 Oct 30 12:33:48.501646 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.499423 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" event={"ID":"2efc5e39-2e4f-4700-96d3-0b478cac830f","Type":"ContainerDied","Data":"ff36dba9d4249848919f3a8857d19202cc4698a9be04c869e07ac61f114e5223"} Oct 30 12:33:48.501646 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.499524 4413 scope.go:122] "RemoveContainer" containerID="db3e8b6e262d9fae95ea47f5f2210d5f9962b0bd0a67bff7ddd4140c529c6546" Oct 30 12:33:48.586881 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.586820 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:33:48.729545 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.729500 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-data\") pod \"2efc5e39-2e4f-4700-96d3-0b478cac830f\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " Oct 30 12:33:48.729767 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.729556 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-token\" (UniqueName: \"kubernetes.io/projected/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-token\") pod \"2efc5e39-2e4f-4700-96d3-0b478cac830f\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " Oct 30 12:33:48.729767 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.729620 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-podinfo\") pod \"2efc5e39-2e4f-4700-96d3-0b478cac830f\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " Oct 30 12:33:48.729767 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.729691 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2efc5e39-2e4f-4700-96d3-0b478cac830f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2efc5e39-2e4f-4700-96d3-0b478cac830f-istiod-ca-cert\") pod \"2efc5e39-2e4f-4700-96d3-0b478cac830f\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " Oct 30 12:33:48.729767 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.729744 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-workload-socket\") pod \"2efc5e39-2e4f-4700-96d3-0b478cac830f\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " Oct 30 12:33:48.729987 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.729809 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-credential-socket\") pod \"2efc5e39-2e4f-4700-96d3-0b478cac830f\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " Oct 30 12:33:48.729987 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.729829 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-data" pod "2efc5e39-2e4f-4700-96d3-0b478cac830f" (UID: "2efc5e39-2e4f-4700-96d3-0b478cac830f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:33:48.729987 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.729858 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2efc5e39-2e4f-4700-96d3-0b478cac830f-kube-api-access-vm6p9\" (UniqueName: \"kubernetes.io/projected/2efc5e39-2e4f-4700-96d3-0b478cac830f-kube-api-access-vm6p9\") pod \"2efc5e39-2e4f-4700-96d3-0b478cac830f\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " Oct 30 12:33:48.729987 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.729896 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-workload-certs\") pod \"2efc5e39-2e4f-4700-96d3-0b478cac830f\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " Oct 30 12:33:48.730178 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.729984 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-envoy\") pod \"2efc5e39-2e4f-4700-96d3-0b478cac830f\" (UID: \"2efc5e39-2e4f-4700-96d3-0b478cac830f\") " Oct 30 12:33:48.730178 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.730131 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-workload-socket" pod "2efc5e39-2e4f-4700-96d3-0b478cac830f" (UID: "2efc5e39-2e4f-4700-96d3-0b478cac830f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:33:48.730312 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.730220 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-workload-certs" pod "2efc5e39-2e4f-4700-96d3-0b478cac830f" (UID: "2efc5e39-2e4f-4700-96d3-0b478cac830f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:33:48.730312 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.730244 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2efc5e39-2e4f-4700-96d3-0b478cac830f-istiod-ca-cert" pod "2efc5e39-2e4f-4700-96d3-0b478cac830f" (UID: "2efc5e39-2e4f-4700-96d3-0b478cac830f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:48.730488 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.730208 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-credential-socket" pod "2efc5e39-2e4f-4700-96d3-0b478cac830f" (UID: "2efc5e39-2e4f-4700-96d3-0b478cac830f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:33:48.730798 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.730650 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2efc5e39-2e4f-4700-96d3-0b478cac830f-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:48.730798 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.730675 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:48.730798 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.730689 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:48.730798 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.730699 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:48.730798 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.730708 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:48.732298 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.732269 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-envoy" pod "2efc5e39-2e4f-4700-96d3-0b478cac830f" (UID: "2efc5e39-2e4f-4700-96d3-0b478cac830f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:33:48.732449 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.732412 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-podinfo" pod "2efc5e39-2e4f-4700-96d3-0b478cac830f" (UID: "2efc5e39-2e4f-4700-96d3-0b478cac830f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:33:48.732449 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.732415 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2efc5e39-2e4f-4700-96d3-0b478cac830f-kube-api-access-vm6p9" pod "2efc5e39-2e4f-4700-96d3-0b478cac830f" (UID: "2efc5e39-2e4f-4700-96d3-0b478cac830f"). InnerVolumeSpecName "kube-api-access-vm6p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:48.732550 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.732484 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-token" pod "2efc5e39-2e4f-4700-96d3-0b478cac830f" (UID: "2efc5e39-2e4f-4700-96d3-0b478cac830f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:48.832519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.832477 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:48.832519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.832511 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:48.832519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.832520 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm6p9\" (UniqueName: \"kubernetes.io/projected/2efc5e39-2e4f-4700-96d3-0b478cac830f-kube-api-access-vm6p9\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:48.832519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:48.832533 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2efc5e39-2e4f-4700-96d3-0b478cac830f-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:49.323465 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:49.323362 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:49.506107 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:49.506073 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" Oct 30 12:33:49.506301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:49.506105 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84" event={"ID":"2efc5e39-2e4f-4700-96d3-0b478cac830f","Type":"ContainerDied","Data":"d5a800939366a0062e35cb0daf30ab82c3f0a3d9e67e0f897b931c84b729eaa7"} Oct 30 12:33:49.506301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:49.506175 4413 scope.go:122] "RemoveContainer" containerID="ff36dba9d4249848919f3a8857d19202cc4698a9be04c869e07ac61f114e5223" Oct 30 12:33:49.570405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:49.570374 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84"] Oct 30 12:33:49.582147 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:49.582076 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-0172c6-openshift-default-6dfcffb6bf-d5s84"] Oct 30 12:33:51.019913 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:51.019883 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2efc5e39-2e4f-4700-96d3-0b478cac830f" path="/var/lib/kubelet/pods/2efc5e39-2e4f-4700-96d3-0b478cac830f/volumes" Oct 30 12:33:51.142977 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:51.142946 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:52.951489 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:52.951454 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:53.424745 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:53.424705 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:54.542433 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:54.542398 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx"] Oct 30 12:33:54.655471 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:54.655435 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:54.769868 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:54.769833 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:56.648977 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:56.648945 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:58.356738 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:58.356698 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:58.471816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:58.471772 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:58.600514 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:58.600436 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:59.128848 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:59.128813 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6"] Oct 30 12:33:59.129067 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:59.129031 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" podUID="a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" containerName="istio-proxy" containerID="cri-o://24658298814e25db7b926dcd5ce51d6174f544ba63d05d0e72468c54183aec8f" gracePeriod=30 Oct 30 12:33:59.542775 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:59.542679 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx" podUID="5cf96915-55fd-47b0-bace-c95cdb790cf2" containerName="payload-processing" containerID="cri-o://99b470f3d937ffc6996ae4ea2ea8845c4d4afadb93a96720dac14240e551b1ec" gracePeriod=25 Oct 30 12:33:59.764284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:59.764217 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:33:59.822388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:59.822357 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx" Oct 30 12:33:59.879740 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:59.879708 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cf96915-55fd-47b0-bace-c95cdb790cf2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf96915-55fd-47b0-bace-c95cdb790cf2-plugins-config-volume\") pod \"5cf96915-55fd-47b0-bace-c95cdb790cf2\" (UID: \"5cf96915-55fd-47b0-bace-c95cdb790cf2\") " Oct 30 12:33:59.879940 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:59.879758 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf96915-55fd-47b0-bace-c95cdb790cf2-kube-api-access-sbp7v\" (UniqueName: \"kubernetes.io/projected/5cf96915-55fd-47b0-bace-c95cdb790cf2-kube-api-access-sbp7v\") pod \"5cf96915-55fd-47b0-bace-c95cdb790cf2\" (UID: \"5cf96915-55fd-47b0-bace-c95cdb790cf2\") " Oct 30 12:33:59.880082 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:59.880060 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf96915-55fd-47b0-bace-c95cdb790cf2-plugins-config-volume" pod "5cf96915-55fd-47b0-bace-c95cdb790cf2" (UID: "5cf96915-55fd-47b0-bace-c95cdb790cf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:33:59.882132 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:59.882103 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf96915-55fd-47b0-bace-c95cdb790cf2-kube-api-access-sbp7v" pod "5cf96915-55fd-47b0-bace-c95cdb790cf2" (UID: "5cf96915-55fd-47b0-bace-c95cdb790cf2"). InnerVolumeSpecName "kube-api-access-sbp7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:33:59.981215 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:59.981186 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf96915-55fd-47b0-bace-c95cdb790cf2-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:33:59.981215 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:33:59.981212 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbp7v\" (UniqueName: \"kubernetes.io/projected/5cf96915-55fd-47b0-bace-c95cdb790cf2-kube-api-access-sbp7v\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:00.132873 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:00.132837 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:00.573181 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:00.573101 4413 generic.go:356] "Generic (PLEG): container finished" podID="5cf96915-55fd-47b0-bace-c95cdb790cf2" containerID="99b470f3d937ffc6996ae4ea2ea8845c4d4afadb93a96720dac14240e551b1ec" exitCode=0 Oct 30 12:34:00.573181 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:00.573157 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx" Oct 30 12:34:00.573729 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:00.573184 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx" event={"ID":"5cf96915-55fd-47b0-bace-c95cdb790cf2","Type":"ContainerDied","Data":"99b470f3d937ffc6996ae4ea2ea8845c4d4afadb93a96720dac14240e551b1ec"} Oct 30 12:34:00.573729 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:00.573240 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx" event={"ID":"5cf96915-55fd-47b0-bace-c95cdb790cf2","Type":"ContainerDied","Data":"81c78cc5525bf7e866cc8488165e6a5ad9ef0fea146c67a05df712cf09d386a0"} Oct 30 12:34:00.573729 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:00.573286 4413 scope.go:122] "RemoveContainer" containerID="99b470f3d937ffc6996ae4ea2ea8845c4d4afadb93a96720dac14240e551b1ec" Oct 30 12:34:00.583497 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:00.583478 4413 scope.go:122] "RemoveContainer" containerID="99b470f3d937ffc6996ae4ea2ea8845c4d4afadb93a96720dac14240e551b1ec" Oct 30 12:34:00.583767 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:34:00.583747 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b470f3d937ffc6996ae4ea2ea8845c4d4afadb93a96720dac14240e551b1ec\": container with ID starting with 99b470f3d937ffc6996ae4ea2ea8845c4d4afadb93a96720dac14240e551b1ec not found: ID does not exist" containerID="99b470f3d937ffc6996ae4ea2ea8845c4d4afadb93a96720dac14240e551b1ec" Oct 30 12:34:00.583816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:00.583776 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b470f3d937ffc6996ae4ea2ea8845c4d4afadb93a96720dac14240e551b1ec"} err="failed to get container status \"99b470f3d937ffc6996ae4ea2ea8845c4d4afadb93a96720dac14240e551b1ec\": rpc error: code = NotFound desc = could not find container \"99b470f3d937ffc6996ae4ea2ea8845c4d4afadb93a96720dac14240e551b1ec\": container with ID starting with 99b470f3d937ffc6996ae4ea2ea8845c4d4afadb93a96720dac14240e551b1ec not found: ID does not exist" Oct 30 12:34:00.643968 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:00.643944 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx"] Oct 30 12:34:00.646466 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:00.646444 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2461e8-865c976f74-8r5vx"] Oct 30 12:34:01.021146 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:01.021111 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cf96915-55fd-47b0-bace-c95cdb790cf2" path="/var/lib/kubelet/pods/5cf96915-55fd-47b0-bace-c95cdb790cf2/volumes" Oct 30 12:34:01.782968 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:01.782939 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:03.184520 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:03.184478 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9"] Oct 30 12:34:03.185525 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:03.185486 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" podUID="a12bc929-1c97-4b32-a01a-250013d307f6" containerName="maas-api" containerID="cri-o://f1abfa75545528a102262ced0232e643ff01d03fa10fbc22086442bfdb3546af" gracePeriod=30 Oct 30 12:34:03.274746 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:03.274710 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:03.668285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:03.667689 4413 generic.go:356] "Generic (PLEG): container finished" podID="a12bc929-1c97-4b32-a01a-250013d307f6" containerID="f1abfa75545528a102262ced0232e643ff01d03fa10fbc22086442bfdb3546af" exitCode=0 Oct 30 12:34:03.668285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:03.667855 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" event={"ID":"a12bc929-1c97-4b32-a01a-250013d307f6","Type":"ContainerDied","Data":"f1abfa75545528a102262ced0232e643ff01d03fa10fbc22086442bfdb3546af"} Oct 30 12:34:04.540272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:04.526744 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-b94h6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 12:34:04.540272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:04.526796 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" podUID="a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 12:34:04.656301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:04.655241 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:34:04.809348 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:04.804454 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a12bc929-1c97-4b32-a01a-250013d307f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12bc929-1c97-4b32-a01a-250013d307f6-maas-api-tls\") pod \"a12bc929-1c97-4b32-a01a-250013d307f6\" (UID: \"a12bc929-1c97-4b32-a01a-250013d307f6\") " Oct 30 12:34:04.809348 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:04.804539 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a12bc929-1c97-4b32-a01a-250013d307f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12bc929-1c97-4b32-a01a-250013d307f6-openshift-service-ca\") pod \"a12bc929-1c97-4b32-a01a-250013d307f6\" (UID: \"a12bc929-1c97-4b32-a01a-250013d307f6\") " Oct 30 12:34:04.809348 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:04.804582 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a12bc929-1c97-4b32-a01a-250013d307f6-kube-api-access-dpc4f\" (UniqueName: \"kubernetes.io/projected/a12bc929-1c97-4b32-a01a-250013d307f6-kube-api-access-dpc4f\") pod \"a12bc929-1c97-4b32-a01a-250013d307f6\" (UID: \"a12bc929-1c97-4b32-a01a-250013d307f6\") " Oct 30 12:34:04.809348 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:04.804796 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a12bc929-1c97-4b32-a01a-250013d307f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12bc929-1c97-4b32-a01a-250013d307f6-metrics-certs\") pod \"a12bc929-1c97-4b32-a01a-250013d307f6\" (UID: \"a12bc929-1c97-4b32-a01a-250013d307f6\") " Oct 30 12:34:04.851281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:04.849454 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a12bc929-1c97-4b32-a01a-250013d307f6-openshift-service-ca" pod "a12bc929-1c97-4b32-a01a-250013d307f6" (UID: "a12bc929-1c97-4b32-a01a-250013d307f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:04.925656 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:04.907951 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12bc929-1c97-4b32-a01a-250013d307f6-kube-api-access-dpc4f" pod "a12bc929-1c97-4b32-a01a-250013d307f6" (UID: "a12bc929-1c97-4b32-a01a-250013d307f6"). InnerVolumeSpecName "kube-api-access-dpc4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:04.930273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:04.929929 4413 generic.go:356] "Generic (PLEG): container finished" podID="a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" containerID="24658298814e25db7b926dcd5ce51d6174f544ba63d05d0e72468c54183aec8f" exitCode=0 Oct 30 12:34:04.930273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:04.930147 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" event={"ID":"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58","Type":"ContainerDied","Data":"24658298814e25db7b926dcd5ce51d6174f544ba63d05d0e72468c54183aec8f"} Oct 30 12:34:04.935100 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:04.934922 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a12bc929-1c97-4b32-a01a-250013d307f6-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:04.935100 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:04.934958 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpc4f\" (UniqueName: \"kubernetes.io/projected/a12bc929-1c97-4b32-a01a-250013d307f6-kube-api-access-dpc4f\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:04.936582 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:04.935888 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" event={"ID":"a12bc929-1c97-4b32-a01a-250013d307f6","Type":"ContainerDied","Data":"1d49a4de6726838a7991cea6fe41707a5517acdf213d86798e07e615acbcbeed"} Oct 30 12:34:04.936582 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:04.935953 4413 scope.go:122] "RemoveContainer" containerID="f1abfa75545528a102262ced0232e643ff01d03fa10fbc22086442bfdb3546af" Oct 30 12:34:04.936582 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:04.936102 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" Oct 30 12:34:04.955322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:04.954083 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a12bc929-1c97-4b32-a01a-250013d307f6-metrics-certs" pod "a12bc929-1c97-4b32-a01a-250013d307f6" (UID: "a12bc929-1c97-4b32-a01a-250013d307f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:34:04.984669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:04.976366 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a12bc929-1c97-4b32-a01a-250013d307f6-maas-api-tls" pod "a12bc929-1c97-4b32-a01a-250013d307f6" (UID: "a12bc929-1c97-4b32-a01a-250013d307f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:34:05.045207 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.045175 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a12bc929-1c97-4b32-a01a-250013d307f6-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:05.047361 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.047331 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a12bc929-1c97-4b32-a01a-250013d307f6-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:05.136291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.131209 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:05.261074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.261046 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:34:05.271923 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.271894 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:05.327273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.320150 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9"] Oct 30 12:34:05.327273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.324619 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9"] Oct 30 12:34:05.386277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.385632 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-workload-certs\") pod \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " Oct 30 12:34:05.386277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.385689 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-workload-socket\") pod \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " Oct 30 12:34:05.386277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.385731 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-token\" (UniqueName: \"kubernetes.io/projected/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-token\") pod \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " Oct 30 12:34:05.386277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.385827 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-envoy\") pod \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " Oct 30 12:34:05.386277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.385922 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-credential-socket\") pod \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " Oct 30 12:34:05.386277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.385975 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-kube-api-access-j6rs6\" (UniqueName: \"kubernetes.io/projected/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-kube-api-access-j6rs6\") pod \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " Oct 30 12:34:05.386277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.386005 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-data\") pod \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " Oct 30 12:34:05.386277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.386042 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istiod-ca-cert\") pod \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " Oct 30 12:34:05.386277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.386136 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-podinfo\") pod \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\" (UID: \"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58\") " Oct 30 12:34:05.387040 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.387012 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-workload-socket" pod "a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" (UID: "a179c4e6-0f4b-4ebb-9d08-f253b8d26e58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:05.387225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.387203 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-workload-certs" pod "a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" (UID: "a179c4e6-0f4b-4ebb-9d08-f253b8d26e58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:05.387446 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.387425 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-data" pod "a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" (UID: "a179c4e6-0f4b-4ebb-9d08-f253b8d26e58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:05.387446 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.387436 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-credential-socket" pod "a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" (UID: "a179c4e6-0f4b-4ebb-9d08-f253b8d26e58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:05.387753 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.387731 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istiod-ca-cert" pod "a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" (UID: "a179c4e6-0f4b-4ebb-9d08-f253b8d26e58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:05.389499 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.389470 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-envoy" pod "a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" (UID: "a179c4e6-0f4b-4ebb-9d08-f253b8d26e58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:05.389847 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.389824 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-podinfo" pod "a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" (UID: "a179c4e6-0f4b-4ebb-9d08-f253b8d26e58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:34:05.390052 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.390035 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-token" pod "a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" (UID: "a179c4e6-0f4b-4ebb-9d08-f253b8d26e58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:05.390344 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.390324 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-kube-api-access-j6rs6" pod "a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" (UID: "a179c4e6-0f4b-4ebb-9d08-f253b8d26e58"). InnerVolumeSpecName "kube-api-access-j6rs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:05.489379 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.489351 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:05.489480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.489385 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6rs6\" (UniqueName: \"kubernetes.io/projected/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-kube-api-access-j6rs6\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:05.489480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.489401 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:05.489480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.489414 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:05.489480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.489427 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:05.489480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.489439 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:05.489480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.489452 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:05.489480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.489464 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:05.489480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.489480 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:05.627973 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.627921 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:05.820458 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:34:05.820367 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:34:05.944043 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.944020 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" Oct 30 12:34:05.944043 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.944040 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6" event={"ID":"a179c4e6-0f4b-4ebb-9d08-f253b8d26e58","Type":"ContainerDied","Data":"f1675eecfb1225e51049dc65ed1480aa6f1644b0d0d41d317b8ddbb4cdc3099a"} Oct 30 12:34:05.944347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:05.944097 4413 scope.go:122] "RemoveContainer" containerID="24658298814e25db7b926dcd5ce51d6174f544ba63d05d0e72468c54183aec8f" Oct 30 12:34:06.002748 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:06.002716 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:06.006442 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:06.006414 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6"] Oct 30 12:34:06.017814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:06.017789 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-b94h6"] Oct 30 12:34:06.124519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:06.124490 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:06.249698 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:06.249662 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:06.368442 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:06.368407 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:06.725601 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:06.725556 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr"] Oct 30 12:34:06.953268 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:06.953215 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:07.028398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:07.028321 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a12bc929-1c97-4b32-a01a-250013d307f6" path="/var/lib/kubelet/pods/a12bc929-1c97-4b32-a01a-250013d307f6/volumes" Oct 30 12:34:07.028821 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:07.028808 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" path="/var/lib/kubelet/pods/a179c4e6-0f4b-4ebb-9d08-f253b8d26e58/volumes" Oct 30 12:34:07.079780 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:07.079746 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:07.468217 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:07.468173 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2461e8-66b5cd7676-w95z9" podUID="a12bc929-1c97-4b32-a01a-250013d307f6" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.136:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:08.656845 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:08.656808 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:09.602815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:09.602782 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:10.356702 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:10.356675 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:11.725965 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:11.725914 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr" podUID="407bfb0f-7219-4558-bfa4-7c31fed7e3e6" containerName="payload-pre-processing" containerID="cri-o://04c6ed8bfc4803b4e28ed298fddabac7632ff25a87c9795d0d056428877a61aa" gracePeriod=25 Oct 30 12:34:11.984379 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:11.984318 4413 generic.go:356] "Generic (PLEG): container finished" podID="407bfb0f-7219-4558-bfa4-7c31fed7e3e6" containerID="04c6ed8bfc4803b4e28ed298fddabac7632ff25a87c9795d0d056428877a61aa" exitCode=0 Oct 30 12:34:11.984379 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:11.984353 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr" event={"ID":"407bfb0f-7219-4558-bfa4-7c31fed7e3e6","Type":"ContainerDied","Data":"04c6ed8bfc4803b4e28ed298fddabac7632ff25a87c9795d0d056428877a61aa"} Oct 30 12:34:11.984520 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:11.984389 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr" event={"ID":"407bfb0f-7219-4558-bfa4-7c31fed7e3e6","Type":"ContainerDied","Data":"69e3f6c99e60346e4ef80d33e62158368024e34f4c294e9c8eb580f035f2f44d"} Oct 30 12:34:11.984520 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:11.984404 4413 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69e3f6c99e60346e4ef80d33e62158368024e34f4c294e9c8eb580f035f2f44d" Oct 30 12:34:12.001687 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:12.001665 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr" Oct 30 12:34:12.076581 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:12.076538 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407bfb0f-7219-4558-bfa4-7c31fed7e3e6-kube-api-access-g6hd4\" (UniqueName: \"kubernetes.io/projected/407bfb0f-7219-4558-bfa4-7c31fed7e3e6-kube-api-access-g6hd4\") pod \"407bfb0f-7219-4558-bfa4-7c31fed7e3e6\" (UID: \"407bfb0f-7219-4558-bfa4-7c31fed7e3e6\") " Oct 30 12:34:12.076770 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:12.076748 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407bfb0f-7219-4558-bfa4-7c31fed7e3e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407bfb0f-7219-4558-bfa4-7c31fed7e3e6-plugins-config-volume\") pod \"407bfb0f-7219-4558-bfa4-7c31fed7e3e6\" (UID: \"407bfb0f-7219-4558-bfa4-7c31fed7e3e6\") " Oct 30 12:34:12.077175 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:12.077147 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407bfb0f-7219-4558-bfa4-7c31fed7e3e6-plugins-config-volume" pod "407bfb0f-7219-4558-bfa4-7c31fed7e3e6" (UID: "407bfb0f-7219-4558-bfa4-7c31fed7e3e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:12.077375 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:12.077358 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407bfb0f-7219-4558-bfa4-7c31fed7e3e6-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:12.078930 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:12.078908 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407bfb0f-7219-4558-bfa4-7c31fed7e3e6-kube-api-access-g6hd4" pod "407bfb0f-7219-4558-bfa4-7c31fed7e3e6" (UID: "407bfb0f-7219-4558-bfa4-7c31fed7e3e6"). InnerVolumeSpecName "kube-api-access-g6hd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:12.100940 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:12.100916 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:12.179319 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:12.179286 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6hd4\" (UniqueName: \"kubernetes.io/projected/407bfb0f-7219-4558-bfa4-7c31fed7e3e6-kube-api-access-g6hd4\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:12.990602 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:12.990572 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr" Oct 30 12:34:13.043183 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:13.043147 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr"] Oct 30 12:34:13.053630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:13.053602 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a124ab-77cb568899-64cfr"] Oct 30 12:34:13.106766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:13.106734 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:13.221881 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:13.221847 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:13.434430 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:13.434402 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw"] Oct 30 12:34:13.434669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:13.434642 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" podUID="c04e3112-1c05-40db-a2ff-659b04c79ffa" containerName="istio-proxy" containerID="cri-o://f8d31e66e3d671a2b3a6f0cc0e858d9bef4fc8471ff47132e71c9e8c8de3d252" gracePeriod=30 Oct 30 12:34:13.485668 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:13.485639 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:14.062164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:14.062116 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:15.018814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:15.018782 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="407bfb0f-7219-4558-bfa4-7c31fed7e3e6" path="/var/lib/kubelet/pods/407bfb0f-7219-4558-bfa4-7c31fed7e3e6/volumes" Oct 30 12:34:15.992566 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:15.992530 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:18.228271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:18.223972 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 12:34:18.228271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:18.226898 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:18.274400 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:18.274360 4413 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Oct 30 12:34:18.638829 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:18.633680 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm"] Oct 30 12:34:18.638829 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:18.633940 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" podUID="638d3046-09df-44a7-864c-2586fe38ec94" containerName="maas-api" containerID="cri-o://feddb8d8a85eda7f73abbd29713fa97f537b8ff05847cc84c5557e6212a1e490" gracePeriod=30 Oct 30 12:34:18.755355 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:18.743572 4413 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:18.802859 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:18.802549 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:19.091538 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.090002 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:34:19.099334 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.095637 4413 generic.go:356] "Generic (PLEG): container finished" podID="c04e3112-1c05-40db-a2ff-659b04c79ffa" containerID="f8d31e66e3d671a2b3a6f0cc0e858d9bef4fc8471ff47132e71c9e8c8de3d252" exitCode=0 Oct 30 12:34:19.099334 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.095800 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" event={"ID":"c04e3112-1c05-40db-a2ff-659b04c79ffa","Type":"ContainerDied","Data":"f8d31e66e3d671a2b3a6f0cc0e858d9bef4fc8471ff47132e71c9e8c8de3d252"} Oct 30 12:34:19.099334 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.095874 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" event={"ID":"c04e3112-1c05-40db-a2ff-659b04c79ffa","Type":"ContainerDied","Data":"7ce3fda7f8084ddf95b5845700da547ec57a95e37253c7d39347696d551d0321"} Oct 30 12:34:19.099334 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.095910 4413 scope.go:122] "RemoveContainer" containerID="f8d31e66e3d671a2b3a6f0cc0e858d9bef4fc8471ff47132e71c9e8c8de3d252" Oct 30 12:34:19.099334 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.096069 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" Oct 30 12:34:19.099334 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.099277 4413 generic.go:356] "Generic (PLEG): container finished" podID="638d3046-09df-44a7-864c-2586fe38ec94" containerID="feddb8d8a85eda7f73abbd29713fa97f537b8ff05847cc84c5557e6212a1e490" exitCode=0 Oct 30 12:34:19.099822 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.099432 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" event={"ID":"638d3046-09df-44a7-864c-2586fe38ec94","Type":"ContainerDied","Data":"feddb8d8a85eda7f73abbd29713fa97f537b8ff05847cc84c5557e6212a1e490"} Oct 30 12:34:19.114487 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.114438 4413 scope.go:122] "RemoveContainer" containerID="67fb2fca7d5938841cde9d6d778cef4d236853fe9d5b72144e811a51046bbe89" Oct 30 12:34:19.132042 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.131614 4413 scope.go:122] "RemoveContainer" containerID="f8d31e66e3d671a2b3a6f0cc0e858d9bef4fc8471ff47132e71c9e8c8de3d252" Oct 30 12:34:19.132602 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:34:19.132226 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d31e66e3d671a2b3a6f0cc0e858d9bef4fc8471ff47132e71c9e8c8de3d252\": container with ID starting with f8d31e66e3d671a2b3a6f0cc0e858d9bef4fc8471ff47132e71c9e8c8de3d252 not found: ID does not exist" containerID="f8d31e66e3d671a2b3a6f0cc0e858d9bef4fc8471ff47132e71c9e8c8de3d252" Oct 30 12:34:19.132932 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.132717 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d31e66e3d671a2b3a6f0cc0e858d9bef4fc8471ff47132e71c9e8c8de3d252"} err="failed to get container status \"f8d31e66e3d671a2b3a6f0cc0e858d9bef4fc8471ff47132e71c9e8c8de3d252\": rpc error: code = NotFound desc = could not find container \"f8d31e66e3d671a2b3a6f0cc0e858d9bef4fc8471ff47132e71c9e8c8de3d252\": container with ID starting with f8d31e66e3d671a2b3a6f0cc0e858d9bef4fc8471ff47132e71c9e8c8de3d252 not found: ID does not exist" Oct 30 12:34:19.132932 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.132748 4413 scope.go:122] "RemoveContainer" containerID="67fb2fca7d5938841cde9d6d778cef4d236853fe9d5b72144e811a51046bbe89" Oct 30 12:34:19.133214 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:34:19.133149 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67fb2fca7d5938841cde9d6d778cef4d236853fe9d5b72144e811a51046bbe89\": container with ID starting with 67fb2fca7d5938841cde9d6d778cef4d236853fe9d5b72144e811a51046bbe89 not found: ID does not exist" containerID="67fb2fca7d5938841cde9d6d778cef4d236853fe9d5b72144e811a51046bbe89" Oct 30 12:34:19.133214 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.133179 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67fb2fca7d5938841cde9d6d778cef4d236853fe9d5b72144e811a51046bbe89"} err="failed to get container status \"67fb2fca7d5938841cde9d6d778cef4d236853fe9d5b72144e811a51046bbe89\": rpc error: code = NotFound desc = could not find container \"67fb2fca7d5938841cde9d6d778cef4d236853fe9d5b72144e811a51046bbe89\": container with ID starting with 67fb2fca7d5938841cde9d6d778cef4d236853fe9d5b72144e811a51046bbe89 not found: ID does not exist" Oct 30 12:34:19.192810 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.192779 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" Oct 30 12:34:19.287853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.286993 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-token\" (UniqueName: \"kubernetes.io/projected/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-token\") pod \"c04e3112-1c05-40db-a2ff-659b04c79ffa\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " Oct 30 12:34:19.287853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.287065 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-podinfo\") pod \"c04e3112-1c05-40db-a2ff-659b04c79ffa\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " Oct 30 12:34:19.287853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.287116 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638d3046-09df-44a7-864c-2586fe38ec94-kube-api-access-hcdlf\" (UniqueName: \"kubernetes.io/projected/638d3046-09df-44a7-864c-2586fe38ec94-kube-api-access-hcdlf\") pod \"638d3046-09df-44a7-864c-2586fe38ec94\" (UID: \"638d3046-09df-44a7-864c-2586fe38ec94\") " Oct 30 12:34:19.287853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.287164 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/638d3046-09df-44a7-864c-2586fe38ec94-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638d3046-09df-44a7-864c-2586fe38ec94-openshift-service-ca\") pod \"638d3046-09df-44a7-864c-2586fe38ec94\" (UID: \"638d3046-09df-44a7-864c-2586fe38ec94\") " Oct 30 12:34:19.287853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.287215 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638d3046-09df-44a7-864c-2586fe38ec94-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638d3046-09df-44a7-864c-2586fe38ec94-maas-api-tls\") pod \"638d3046-09df-44a7-864c-2586fe38ec94\" (UID: \"638d3046-09df-44a7-864c-2586fe38ec94\") " Oct 30 12:34:19.287853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.287300 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-workload-certs\") pod \"c04e3112-1c05-40db-a2ff-659b04c79ffa\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " Oct 30 12:34:19.287853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.287341 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-workload-socket\") pod \"c04e3112-1c05-40db-a2ff-659b04c79ffa\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " Oct 30 12:34:19.287853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.287372 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04e3112-1c05-40db-a2ff-659b04c79ffa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04e3112-1c05-40db-a2ff-659b04c79ffa-istiod-ca-cert\") pod \"c04e3112-1c05-40db-a2ff-659b04c79ffa\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " Oct 30 12:34:19.287853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.287405 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-envoy\") pod \"c04e3112-1c05-40db-a2ff-659b04c79ffa\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " Oct 30 12:34:19.287853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.287523 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638d3046-09df-44a7-864c-2586fe38ec94-metrics-certs\" (UniqueName: \"kubernetes.io/secret/638d3046-09df-44a7-864c-2586fe38ec94-metrics-certs\") pod \"638d3046-09df-44a7-864c-2586fe38ec94\" (UID: \"638d3046-09df-44a7-864c-2586fe38ec94\") " Oct 30 12:34:19.287853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.287579 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-credential-socket\") pod \"c04e3112-1c05-40db-a2ff-659b04c79ffa\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " Oct 30 12:34:19.287853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.287649 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04e3112-1c05-40db-a2ff-659b04c79ffa-kube-api-access-dkxhj\" (UniqueName: \"kubernetes.io/projected/c04e3112-1c05-40db-a2ff-659b04c79ffa-kube-api-access-dkxhj\") pod \"c04e3112-1c05-40db-a2ff-659b04c79ffa\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " Oct 30 12:34:19.287853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.287677 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-data\") pod \"c04e3112-1c05-40db-a2ff-659b04c79ffa\" (UID: \"c04e3112-1c05-40db-a2ff-659b04c79ffa\") " Oct 30 12:34:19.289108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.288178 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-workload-certs" pod "c04e3112-1c05-40db-a2ff-659b04c79ffa" (UID: "c04e3112-1c05-40db-a2ff-659b04c79ffa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:19.289108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.288859 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:19.289108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.289076 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-workload-socket" pod "c04e3112-1c05-40db-a2ff-659b04c79ffa" (UID: "c04e3112-1c05-40db-a2ff-659b04c79ffa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:19.289381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.289360 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04e3112-1c05-40db-a2ff-659b04c79ffa-istiod-ca-cert" pod "c04e3112-1c05-40db-a2ff-659b04c79ffa" (UID: "c04e3112-1c05-40db-a2ff-659b04c79ffa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:19.294869 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.294622 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/638d3046-09df-44a7-864c-2586fe38ec94-openshift-service-ca" pod "638d3046-09df-44a7-864c-2586fe38ec94" (UID: "638d3046-09df-44a7-864c-2586fe38ec94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:19.295658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.295326 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-credential-socket" pod "c04e3112-1c05-40db-a2ff-659b04c79ffa" (UID: "c04e3112-1c05-40db-a2ff-659b04c79ffa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:19.296302 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.296238 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-data" pod "c04e3112-1c05-40db-a2ff-659b04c79ffa" (UID: "c04e3112-1c05-40db-a2ff-659b04c79ffa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:19.297390 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.297364 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-podinfo" pod "c04e3112-1c05-40db-a2ff-659b04c79ffa" (UID: "c04e3112-1c05-40db-a2ff-659b04c79ffa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:34:19.303815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.303427 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638d3046-09df-44a7-864c-2586fe38ec94-kube-api-access-hcdlf" pod "638d3046-09df-44a7-864c-2586fe38ec94" (UID: "638d3046-09df-44a7-864c-2586fe38ec94"). InnerVolumeSpecName "kube-api-access-hcdlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:19.303815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.303592 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04e3112-1c05-40db-a2ff-659b04c79ffa-kube-api-access-dkxhj" pod "c04e3112-1c05-40db-a2ff-659b04c79ffa" (UID: "c04e3112-1c05-40db-a2ff-659b04c79ffa"). InnerVolumeSpecName "kube-api-access-dkxhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:19.304494 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.304472 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638d3046-09df-44a7-864c-2586fe38ec94-metrics-certs" pod "638d3046-09df-44a7-864c-2586fe38ec94" (UID: "638d3046-09df-44a7-864c-2586fe38ec94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:34:19.304643 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.304472 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-token" pod "c04e3112-1c05-40db-a2ff-659b04c79ffa" (UID: "c04e3112-1c05-40db-a2ff-659b04c79ffa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:19.304794 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.304760 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-envoy" pod "c04e3112-1c05-40db-a2ff-659b04c79ffa" (UID: "c04e3112-1c05-40db-a2ff-659b04c79ffa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:19.306145 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.306124 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638d3046-09df-44a7-864c-2586fe38ec94-maas-api-tls" pod "638d3046-09df-44a7-864c-2586fe38ec94" (UID: "638d3046-09df-44a7-864c-2586fe38ec94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:34:19.390074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.390038 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/638d3046-09df-44a7-864c-2586fe38ec94-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:19.390074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.390064 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/638d3046-09df-44a7-864c-2586fe38ec94-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:19.390074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.390073 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:19.390074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.390081 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04e3112-1c05-40db-a2ff-659b04c79ffa-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:19.390371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.390089 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:19.390371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.390096 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/638d3046-09df-44a7-864c-2586fe38ec94-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:19.390371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.390103 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:19.390371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.390111 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkxhj\" (UniqueName: \"kubernetes.io/projected/c04e3112-1c05-40db-a2ff-659b04c79ffa-kube-api-access-dkxhj\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:19.390371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.390119 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:19.390371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.390127 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:19.390371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.390134 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04e3112-1c05-40db-a2ff-659b04c79ffa-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:19.390371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.390141 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcdlf\" (UniqueName: \"kubernetes.io/projected/638d3046-09df-44a7-864c-2586fe38ec94-kube-api-access-hcdlf\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:19.441793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.441767 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw"] Oct 30 12:34:19.445090 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:19.445066 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw"] Oct 30 12:34:20.062624 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:20.062588 4413 patch_prober.go:29] interesting pod/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:34:20.062794 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:20.062653 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-2461e8-openshift-default-5f794f99df-hwhkw" podUID="c04e3112-1c05-40db-a2ff-659b04c79ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:20.108553 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:20.108527 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" Oct 30 12:34:20.108724 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:20.108518 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm" event={"ID":"638d3046-09df-44a7-864c-2586fe38ec94","Type":"ContainerDied","Data":"45255fe0ceb66783670e379d3f83b307a1a9f38bb565c582988b221d0849fa2a"} Oct 30 12:34:20.108724 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:20.108685 4413 scope.go:122] "RemoveContainer" containerID="feddb8d8a85eda7f73abbd29713fa97f537b8ff05847cc84c5557e6212a1e490" Oct 30 12:34:20.159971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:20.159944 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm"] Oct 30 12:34:20.163396 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:20.163374 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a124ab-d7f87df6b-xqxkm"] Oct 30 12:34:20.499816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:20.499786 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:20.621682 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:20.621651 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:20.738544 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:20.738512 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:21.020194 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:21.020161 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="638d3046-09df-44a7-864c-2586fe38ec94" path="/var/lib/kubelet/pods/638d3046-09df-44a7-864c-2586fe38ec94/volumes" Oct 30 12:34:21.020686 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:21.020671 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04e3112-1c05-40db-a2ff-659b04c79ffa" path="/var/lib/kubelet/pods/c04e3112-1c05-40db-a2ff-659b04c79ffa/volumes" Oct 30 12:34:21.124665 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:21.124620 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:22.284129 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:22.284094 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t"] Oct 30 12:34:22.396466 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:22.396430 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:22.526500 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:22.526453 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:24.054415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:24.054384 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:25.744878 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:25.744850 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:27.285011 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:27.284958 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t" podUID="06361f15-8bef-4b59-9b23-89c04c941b65" containerName="payload-processing" containerID="cri-o://69605c775f6d078ac85b27d68db3b83938b3967a7d935f08287f119a8c324998" gracePeriod=25 Oct 30 12:34:27.453131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:27.453098 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:27.564985 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:27.564964 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t" Oct 30 12:34:27.686586 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:27.686544 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:27.700345 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:27.700319 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06361f15-8bef-4b59-9b23-89c04c941b65-kube-api-access-6vr6j\" (UniqueName: \"kubernetes.io/projected/06361f15-8bef-4b59-9b23-89c04c941b65-kube-api-access-6vr6j\") pod \"06361f15-8bef-4b59-9b23-89c04c941b65\" (UID: \"06361f15-8bef-4b59-9b23-89c04c941b65\") " Oct 30 12:34:27.700493 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:27.700407 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06361f15-8bef-4b59-9b23-89c04c941b65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06361f15-8bef-4b59-9b23-89c04c941b65-plugins-config-volume\") pod \"06361f15-8bef-4b59-9b23-89c04c941b65\" (UID: \"06361f15-8bef-4b59-9b23-89c04c941b65\") " Oct 30 12:34:27.700819 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:27.700796 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06361f15-8bef-4b59-9b23-89c04c941b65-plugins-config-volume" pod "06361f15-8bef-4b59-9b23-89c04c941b65" (UID: "06361f15-8bef-4b59-9b23-89c04c941b65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:27.700999 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:27.700980 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06361f15-8bef-4b59-9b23-89c04c941b65-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:27.702643 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:27.702618 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06361f15-8bef-4b59-9b23-89c04c941b65-kube-api-access-6vr6j" pod "06361f15-8bef-4b59-9b23-89c04c941b65" (UID: "06361f15-8bef-4b59-9b23-89c04c941b65"). InnerVolumeSpecName "kube-api-access-6vr6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:27.799571 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:27.799488 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:27.802006 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:27.801978 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vr6j\" (UniqueName: \"kubernetes.io/projected/06361f15-8bef-4b59-9b23-89c04c941b65-kube-api-access-6vr6j\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:27.996839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:27.996800 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:28.018612 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.018582 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf"] Oct 30 12:34:28.018872 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.018843 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" podUID="db669f7d-1af0-4f6d-9a65-f3295751ceb8" containerName="istio-proxy" containerID="cri-o://8dd2439f66f5bed3dd7c75c697cca5d737550a2abb9e1e0813bd3fea66784228" gracePeriod=30 Oct 30 12:34:28.113008 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.112925 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:28.183791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.183754 4413 generic.go:356] "Generic (PLEG): container finished" podID="06361f15-8bef-4b59-9b23-89c04c941b65" containerID="69605c775f6d078ac85b27d68db3b83938b3967a7d935f08287f119a8c324998" exitCode=0 Oct 30 12:34:28.184157 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.184128 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t" event={"ID":"06361f15-8bef-4b59-9b23-89c04c941b65","Type":"ContainerDied","Data":"69605c775f6d078ac85b27d68db3b83938b3967a7d935f08287f119a8c324998"} Oct 30 12:34:28.184372 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.184193 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t" event={"ID":"06361f15-8bef-4b59-9b23-89c04c941b65","Type":"ContainerDied","Data":"ba68fd5562a404c18280db679d84eb5450fa706654cdd781b160f87c3b86d520"} Oct 30 12:34:28.184372 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.184222 4413 scope.go:122] "RemoveContainer" containerID="69605c775f6d078ac85b27d68db3b83938b3967a7d935f08287f119a8c324998" Oct 30 12:34:28.184500 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.184412 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t" Oct 30 12:34:28.208932 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.208903 4413 scope.go:122] "RemoveContainer" containerID="69605c775f6d078ac85b27d68db3b83938b3967a7d935f08287f119a8c324998" Oct 30 12:34:28.209332 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:34:28.209307 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69605c775f6d078ac85b27d68db3b83938b3967a7d935f08287f119a8c324998\": container with ID starting with 69605c775f6d078ac85b27d68db3b83938b3967a7d935f08287f119a8c324998 not found: ID does not exist" containerID="69605c775f6d078ac85b27d68db3b83938b3967a7d935f08287f119a8c324998" Oct 30 12:34:28.209447 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.209342 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69605c775f6d078ac85b27d68db3b83938b3967a7d935f08287f119a8c324998"} err="failed to get container status \"69605c775f6d078ac85b27d68db3b83938b3967a7d935f08287f119a8c324998\": rpc error: code = NotFound desc = could not find container \"69605c775f6d078ac85b27d68db3b83938b3967a7d935f08287f119a8c324998\": container with ID starting with 69605c775f6d078ac85b27d68db3b83938b3967a7d935f08287f119a8c324998 not found: ID does not exist" Oct 30 12:34:28.240433 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.240406 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t"] Oct 30 12:34:28.244173 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.244153 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5ce7c4-8b7467cc6-gtf5t"] Oct 30 12:34:28.606632 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.606600 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2"] Oct 30 12:34:28.607544 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607521 4413 cpu_manager.go:414] "containerMap: removing container" podUID="2efc5e39-2e4f-4700-96d3-0b478cac830f" containerName="istio-proxy" Oct 30 12:34:28.607613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607546 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2efc5e39-2e4f-4700-96d3-0b478cac830f" containerName="istio-proxy" Oct 30 12:34:28.607613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607562 4413 cpu_manager.go:414] "containerMap: removing container" podUID="64cfb9ee-656e-4d1b-af81-1a73742935b7" containerName="maas-api" Oct 30 12:34:28.607613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607570 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64cfb9ee-656e-4d1b-af81-1a73742935b7" containerName="maas-api" Oct 30 12:34:28.607613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607590 4413 cpu_manager.go:414] "containerMap: removing container" podUID="a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" containerName="istio-proxy" Oct 30 12:34:28.607613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607597 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" containerName="istio-proxy" Oct 30 12:34:28.607613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607610 4413 cpu_manager.go:414] "containerMap: removing container" podUID="407bfb0f-7219-4558-bfa4-7c31fed7e3e6" containerName="payload-pre-processing" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607617 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="407bfb0f-7219-4558-bfa4-7c31fed7e3e6" containerName="payload-pre-processing" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607638 4413 cpu_manager.go:414] "containerMap: removing container" podUID="9d96d03b-bb3c-4841-a9e5-94996131da71" containerName="payload-processing" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607645 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d96d03b-bb3c-4841-a9e5-94996131da71" containerName="payload-processing" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607654 4413 cpu_manager.go:414] "containerMap: removing container" podUID="95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6" containerName="payload-pre-processing" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607662 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6" containerName="payload-pre-processing" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607676 4413 cpu_manager.go:414] "containerMap: removing container" podUID="5cf96915-55fd-47b0-bace-c95cdb790cf2" containerName="payload-processing" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607683 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf96915-55fd-47b0-bace-c95cdb790cf2" containerName="payload-processing" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607705 4413 cpu_manager.go:414] "containerMap: removing container" podUID="c04e3112-1c05-40db-a2ff-659b04c79ffa" containerName="istio-proxy" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607713 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04e3112-1c05-40db-a2ff-659b04c79ffa" containerName="istio-proxy" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607723 4413 cpu_manager.go:414] "containerMap: removing container" podUID="b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9" containerName="payload-processing" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607731 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9" containerName="payload-processing" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607763 4413 cpu_manager.go:414] "containerMap: removing container" podUID="06361f15-8bef-4b59-9b23-89c04c941b65" containerName="payload-processing" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607770 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06361f15-8bef-4b59-9b23-89c04c941b65" containerName="payload-processing" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607788 4413 cpu_manager.go:414] "containerMap: removing container" podUID="969651c2-7e9e-4b32-8e86-be72e29bd06b" containerName="main" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607795 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969651c2-7e9e-4b32-8e86-be72e29bd06b" containerName="main" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607809 4413 cpu_manager.go:414] "containerMap: removing container" podUID="a12bc929-1c97-4b32-a01a-250013d307f6" containerName="maas-api" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607816 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a12bc929-1c97-4b32-a01a-250013d307f6" containerName="maas-api" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607828 4413 cpu_manager.go:414] "containerMap: removing container" podUID="638d3046-09df-44a7-864c-2586fe38ec94" containerName="maas-api" Oct 30 12:34:28.607898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607835 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638d3046-09df-44a7-864c-2586fe38ec94" containerName="maas-api" Oct 30 12:34:28.608914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.607979 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="2efc5e39-2e4f-4700-96d3-0b478cac830f" containerName="istio-proxy" Oct 30 12:34:28.608914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.608005 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="64cfb9ee-656e-4d1b-af81-1a73742935b7" containerName="maas-api" Oct 30 12:34:28.608914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.608018 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="a179c4e6-0f4b-4ebb-9d08-f253b8d26e58" containerName="istio-proxy" Oct 30 12:34:28.608914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.608029 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="407bfb0f-7219-4558-bfa4-7c31fed7e3e6" containerName="payload-pre-processing" Oct 30 12:34:28.608914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.608042 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d96d03b-bb3c-4841-a9e5-94996131da71" containerName="payload-processing" Oct 30 12:34:28.608914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.608051 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="95f0f90e-ba2f-46e4-8788-cc6a9c9d47e6" containerName="payload-pre-processing" Oct 30 12:34:28.608914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.608062 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf96915-55fd-47b0-bace-c95cdb790cf2" containerName="payload-processing" Oct 30 12:34:28.608914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.608071 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04e3112-1c05-40db-a2ff-659b04c79ffa" containerName="istio-proxy" Oct 30 12:34:28.608914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.608079 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="2efc5e39-2e4f-4700-96d3-0b478cac830f" containerName="istio-proxy" Oct 30 12:34:28.608914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.608086 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b0d9a1-da2a-46ee-a016-0993d8fc4cb9" containerName="payload-processing" Oct 30 12:34:28.608914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.608097 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="06361f15-8bef-4b59-9b23-89c04c941b65" containerName="payload-processing" Oct 30 12:34:28.608914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.608108 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="969651c2-7e9e-4b32-8e86-be72e29bd06b" containerName="main" Oct 30 12:34:28.608914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.608116 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="a12bc929-1c97-4b32-a01a-250013d307f6" containerName="maas-api" Oct 30 12:34:28.608914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.608125 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="638d3046-09df-44a7-864c-2586fe38ec94" containerName="maas-api" Oct 30 12:34:28.613758 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.613738 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.633457 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.633432 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2"] Oct 30 12:34:28.634907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.634888 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:28.641926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.641907 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-2675c0-openshift-default-dockercfg-2rcjh\"" Oct 30 12:34:28.712938 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.712838 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-workload-certs\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.713305 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.713224 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-podinfo\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.713513 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.713324 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-envoy\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.713513 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.713406 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7sgl\" (UniqueName: \"kubernetes.io/projected/75e43f0b-df10-4b4e-b7af-67d515178a84-kube-api-access-z7sgl\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.713513 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.713440 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-workload-socket\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.713513 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.713477 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-token\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.713968 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.713514 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75e43f0b-df10-4b4e-b7af-67d515178a84-istiod-ca-cert\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.713968 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.713608 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-data\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.713968 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.713638 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-credential-socket\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.750990 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.750874 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:28.815356 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.815325 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-data\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.815356 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.815365 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-credential-socket\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.815609 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.815523 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-workload-certs\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.815609 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.815567 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-podinfo\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.815609 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.815594 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-envoy\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.815749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.815665 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7sgl\" (UniqueName: \"kubernetes.io/projected/75e43f0b-df10-4b4e-b7af-67d515178a84-kube-api-access-z7sgl\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.815749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.815696 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-workload-socket\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.815749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.815728 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-token\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.815922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.815801 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-data\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.815990 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.815928 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-credential-socket\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.816049 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.816015 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75e43f0b-df10-4b4e-b7af-67d515178a84-istiod-ca-cert\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.816388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.816340 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-workload-certs\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.816539 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.816502 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-workload-socket\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.816919 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.816896 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75e43f0b-df10-4b4e-b7af-67d515178a84-istiod-ca-cert\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.819362 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.819102 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-podinfo\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.821185 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.821138 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-envoy\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.838693 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.838666 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-token\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.838856 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.838802 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7sgl\" (UniqueName: \"kubernetes.io/projected/75e43f0b-df10-4b4e-b7af-67d515178a84-kube-api-access-z7sgl\") pod \"e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:28.872740 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.872668 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:28.928302 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:28.928234 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:29.020987 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.020959 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06361f15-8bef-4b59-9b23-89c04c941b65" path="/var/lib/kubelet/pods/06361f15-8bef-4b59-9b23-89c04c941b65/volumes" Oct 30 12:34:29.248005 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.247971 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq"] Oct 30 12:34:29.248840 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.248818 4413 cpu_manager.go:414] "containerMap: removing container" podUID="c04e3112-1c05-40db-a2ff-659b04c79ffa" containerName="istio-proxy" Oct 30 12:34:29.248915 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.248844 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04e3112-1c05-40db-a2ff-659b04c79ffa" containerName="istio-proxy" Oct 30 12:34:29.248969 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.248917 4413 cpu_manager.go:414] "containerMap: removing container" podUID="2efc5e39-2e4f-4700-96d3-0b478cac830f" containerName="istio-proxy" Oct 30 12:34:29.248969 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.248925 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2efc5e39-2e4f-4700-96d3-0b478cac830f" containerName="istio-proxy" Oct 30 12:34:29.249089 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.249076 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04e3112-1c05-40db-a2ff-659b04c79ffa" containerName="istio-proxy" Oct 30 12:34:29.254391 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.254369 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.272017 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.271991 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-812990-openshift-default-dockercfg-kpslv\"" Oct 30 12:34:29.273943 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.273918 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq"] Oct 30 12:34:29.280168 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:34:29.280144 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e43f0b_df10_4b4e_b7af_67d515178a84.slice/crio-806529ba39b622e2cac8a837ced95f7c69b52c66a61513f9f32e32b47aff4707 WatchSource:0}: Error finding container 806529ba39b622e2cac8a837ced95f7c69b52c66a61513f9f32e32b47aff4707: Status 404 returned error can't find the container with id 806529ba39b622e2cac8a837ced95f7c69b52c66a61513f9f32e32b47aff4707 Oct 30 12:34:29.283968 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.283941 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:34:29.284208 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.284168 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:34:29.284394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.284235 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:34:29.285942 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.285915 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2"] Oct 30 12:34:29.305529 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.305504 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:29.323747 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.323721 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsgzm\" (UniqueName: \"kubernetes.io/projected/5b66ed77-d0a7-4180-ba18-8598280715af-kube-api-access-dsgzm\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.323846 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.323764 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-credential-socket\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.323906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.323888 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-workload-certs\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.323944 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.323928 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-istio-data\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.323988 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.323957 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b66ed77-d0a7-4180-ba18-8598280715af-istiod-ca-cert\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.324080 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.324062 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b66ed77-d0a7-4180-ba18-8598280715af-istio-token\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.324121 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.324091 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-istio-envoy\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.324164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.324128 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b66ed77-d0a7-4180-ba18-8598280715af-istio-podinfo\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.324240 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.324223 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-workload-socket\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.426124 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.426089 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-workload-socket\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.426342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.426193 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsgzm\" (UniqueName: \"kubernetes.io/projected/5b66ed77-d0a7-4180-ba18-8598280715af-kube-api-access-dsgzm\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.426342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.426227 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-credential-socket\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.426342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.426332 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-workload-certs\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.426534 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.426378 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-istio-data\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.426534 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.426424 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b66ed77-d0a7-4180-ba18-8598280715af-istiod-ca-cert\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.426534 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.426494 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-workload-socket\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.426534 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.426506 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b66ed77-d0a7-4180-ba18-8598280715af-istio-token\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.426759 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.426542 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-istio-envoy\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.426759 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.426587 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b66ed77-d0a7-4180-ba18-8598280715af-istio-podinfo\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.426759 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.426729 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-credential-socket\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.427125 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.427096 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-workload-certs\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.427209 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.427187 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-istio-data\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.427935 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.427904 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b66ed77-d0a7-4180-ba18-8598280715af-istiod-ca-cert\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.429130 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.429107 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-istio-envoy\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.429226 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.429209 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b66ed77-d0a7-4180-ba18-8598280715af-istio-podinfo\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.449795 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.449760 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b66ed77-d0a7-4180-ba18-8598280715af-istio-token\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.449930 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.449825 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsgzm\" (UniqueName: \"kubernetes.io/projected/5b66ed77-d0a7-4180-ba18-8598280715af-kube-api-access-dsgzm\") pod \"e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.527041 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.526968 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:29.569900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.569868 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:29.648073 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.648046 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:29.889721 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.889689 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq"] Oct 30 12:34:29.891357 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:34:29.891331 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b66ed77_d0a7_4180_ba18_8598280715af.slice/crio-23c8a49e2dbf1f5c73e1bd1d17a02f543ce4e06254715c9bf52aaa857b56718f WatchSource:0}: Error finding container 23c8a49e2dbf1f5c73e1bd1d17a02f543ce4e06254715c9bf52aaa857b56718f: Status 404 returned error can't find the container with id 23c8a49e2dbf1f5c73e1bd1d17a02f543ce4e06254715c9bf52aaa857b56718f Oct 30 12:34:29.893921 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.893882 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:34:29.894043 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.893968 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:34:29.894117 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:29.894046 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:34:30.220471 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:30.220370 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" event={"ID":"75e43f0b-df10-4b4e-b7af-67d515178a84","Type":"ContainerStarted","Data":"e1f4a8c880313b406b4cd2b36e5344c75bdc8e7408a619bb8f538930f572d732"} Oct 30 12:34:30.220471 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:30.220434 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" event={"ID":"75e43f0b-df10-4b4e-b7af-67d515178a84","Type":"ContainerStarted","Data":"806529ba39b622e2cac8a837ced95f7c69b52c66a61513f9f32e32b47aff4707"} Oct 30 12:34:30.222199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:30.222172 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" event={"ID":"5b66ed77-d0a7-4180-ba18-8598280715af","Type":"ContainerStarted","Data":"d5eb27e80038c448c69324e58fde9eb52845e69e86953a36445d70c517601905"} Oct 30 12:34:30.222354 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:30.222220 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" event={"ID":"5b66ed77-d0a7-4180-ba18-8598280715af","Type":"ContainerStarted","Data":"23c8a49e2dbf1f5c73e1bd1d17a02f543ce4e06254715c9bf52aaa857b56718f"} Oct 30 12:34:30.266215 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:30.266158 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" podStartSLOduration=2.2661444250000002 podStartE2EDuration="2.266144425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:30.263904606 +0000 UTC m=+2423.373747510" watchObservedRunningTime="2026-10-30 12:34:30.266144425 +0000 UTC m=+2423.375987305" Oct 30 12:34:30.309457 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:30.309410 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" podStartSLOduration=1.309396069 podStartE2EDuration="1.309396069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:30.306332097 +0000 UTC m=+2423.416174977" watchObservedRunningTime="2026-10-30 12:34:30.309396069 +0000 UTC m=+2423.419239030" Oct 30 12:34:30.570440 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:30.570350 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:30.575961 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:30.575936 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:30.929183 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:30.929157 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:30.934430 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:30.934407 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:31.228543 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:31.228459 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:31.228753 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:31.228730 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:31.229321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:31.229300 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:34:31.229452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:31.229363 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:34:31.496753 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:31.496639 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:32.996138 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:32.996105 4413 patch_prober.go:29] interesting pod/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:34:32.996541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:32.996159 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" podUID="db669f7d-1af0-4f6d-9a65-f3295751ceb8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:34:33.163185 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.163150 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:33.246791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.246709 4413 generic.go:356] "Generic (PLEG): container finished" podID="db669f7d-1af0-4f6d-9a65-f3295751ceb8" containerID="8dd2439f66f5bed3dd7c75c697cca5d737550a2abb9e1e0813bd3fea66784228" exitCode=0 Oct 30 12:34:33.246967 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.246779 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" event={"ID":"db669f7d-1af0-4f6d-9a65-f3295751ceb8","Type":"ContainerDied","Data":"8dd2439f66f5bed3dd7c75c697cca5d737550a2abb9e1e0813bd3fea66784228"} Oct 30 12:34:33.246967 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.246848 4413 scope.go:122] "RemoveContainer" containerID="9c5df2ade0ea8f11cc2fbb19bdc805d5bff57ab62982e9957090a760284dc47f" Oct 30 12:34:33.312113 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.312092 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:34:33.380535 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.380499 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-credential-socket\") pod \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " Oct 30 12:34:33.380714 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.380551 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-workload-certs\") pod \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " Oct 30 12:34:33.380714 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.380585 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-podinfo\") pod \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " Oct 30 12:34:33.380714 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.380656 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-token\" (UniqueName: \"kubernetes.io/projected/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-token\") pod \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " Oct 30 12:34:33.380891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.380724 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-data\") pod \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " Oct 30 12:34:33.380891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.380770 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db669f7d-1af0-4f6d-9a65-f3295751ceb8-kube-api-access-bk5v2\" (UniqueName: \"kubernetes.io/projected/db669f7d-1af0-4f6d-9a65-f3295751ceb8-kube-api-access-bk5v2\") pod \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " Oct 30 12:34:33.380891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.380855 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-envoy\") pod \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " Oct 30 12:34:33.381046 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.380891 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-workload-socket\") pod \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " Oct 30 12:34:33.381046 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.380934 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istiod-ca-cert\") pod \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\" (UID: \"db669f7d-1af0-4f6d-9a65-f3295751ceb8\") " Oct 30 12:34:33.381165 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.381138 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-workload-certs" pod "db669f7d-1af0-4f6d-9a65-f3295751ceb8" (UID: "db669f7d-1af0-4f6d-9a65-f3295751ceb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:33.381223 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.381206 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-credential-socket" pod "db669f7d-1af0-4f6d-9a65-f3295751ceb8" (UID: "db669f7d-1af0-4f6d-9a65-f3295751ceb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:33.381650 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.381608 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istiod-ca-cert" pod "db669f7d-1af0-4f6d-9a65-f3295751ceb8" (UID: "db669f7d-1af0-4f6d-9a65-f3295751ceb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:33.381822 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.381803 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:33.381900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.381828 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:33.381900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.381845 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:33.382221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.382145 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-workload-socket" pod "db669f7d-1af0-4f6d-9a65-f3295751ceb8" (UID: "db669f7d-1af0-4f6d-9a65-f3295751ceb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:33.382688 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.382662 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-data" pod "db669f7d-1af0-4f6d-9a65-f3295751ceb8" (UID: "db669f7d-1af0-4f6d-9a65-f3295751ceb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:33.384201 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.384169 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-envoy" pod "db669f7d-1af0-4f6d-9a65-f3295751ceb8" (UID: "db669f7d-1af0-4f6d-9a65-f3295751ceb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:33.384562 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.384538 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db669f7d-1af0-4f6d-9a65-f3295751ceb8-kube-api-access-bk5v2" pod "db669f7d-1af0-4f6d-9a65-f3295751ceb8" (UID: "db669f7d-1af0-4f6d-9a65-f3295751ceb8"). InnerVolumeSpecName "kube-api-access-bk5v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:33.384786 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.384760 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-podinfo" pod "db669f7d-1af0-4f6d-9a65-f3295751ceb8" (UID: "db669f7d-1af0-4f6d-9a65-f3295751ceb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:34:33.384857 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.384805 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-token" pod "db669f7d-1af0-4f6d-9a65-f3295751ceb8" (UID: "db669f7d-1af0-4f6d-9a65-f3295751ceb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:33.483685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.483648 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:33.483685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.483679 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk5v2\" (UniqueName: \"kubernetes.io/projected/db669f7d-1af0-4f6d-9a65-f3295751ceb8-kube-api-access-bk5v2\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:33.483685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.483689 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:33.483685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.483697 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db669f7d-1af0-4f6d-9a65-f3295751ceb8-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:33.483960 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.483705 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:33.483960 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.483713 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db669f7d-1af0-4f6d-9a65-f3295751ceb8-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:33.533936 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.533852 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:33.649813 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.649776 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:33.785401 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.785149 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:33.904046 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:33.904017 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:34.254199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.254169 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" event={"ID":"db669f7d-1af0-4f6d-9a65-f3295751ceb8","Type":"ContainerDied","Data":"d67e1e3cc2051b0b1f6165318b491eb682c917d1ce63c55f8574b06eb2862876"} Oct 30 12:34:34.254199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.254198 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf" Oct 30 12:34:34.254727 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.254226 4413 scope.go:122] "RemoveContainer" containerID="8dd2439f66f5bed3dd7c75c697cca5d737550a2abb9e1e0813bd3fea66784228" Oct 30 12:34:34.327831 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.327801 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8"] Oct 30 12:34:34.328405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.328391 4413 cpu_manager.go:414] "containerMap: removing container" podUID="db669f7d-1af0-4f6d-9a65-f3295751ceb8" containerName="istio-proxy" Oct 30 12:34:34.328459 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.328406 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db669f7d-1af0-4f6d-9a65-f3295751ceb8" containerName="istio-proxy" Oct 30 12:34:34.328459 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.328418 4413 cpu_manager.go:414] "containerMap: removing container" podUID="db669f7d-1af0-4f6d-9a65-f3295751ceb8" containerName="istio-proxy" Oct 30 12:34:34.328459 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.328422 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db669f7d-1af0-4f6d-9a65-f3295751ceb8" containerName="istio-proxy" Oct 30 12:34:34.328580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.328509 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="db669f7d-1af0-4f6d-9a65-f3295751ceb8" containerName="istio-proxy" Oct 30 12:34:34.328580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.328518 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="db669f7d-1af0-4f6d-9a65-f3295751ceb8" containerName="istio-proxy" Oct 30 12:34:34.333319 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.333296 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.343532 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.343507 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf"] Oct 30 12:34:34.351923 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.351905 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-0bda67-openshift-default-dockercfg-7n4kl\"" Oct 30 12:34:34.354560 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.354539 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-a124ab-openshift-default-79c9dc9fdf-vkzqf"] Oct 30 12:34:34.360701 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.360673 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8"] Oct 30 12:34:34.394356 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.394326 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-credential-socket\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.394519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.394366 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6435f44e-980e-4166-868b-ff94fa2010b9-istiod-ca-cert\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.394519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.394403 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzrm2\" (UniqueName: \"kubernetes.io/projected/6435f44e-980e-4166-868b-ff94fa2010b9-kube-api-access-vzrm2\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.394608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.394579 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-istio-data\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.394734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.394715 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-workload-socket\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.394793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.394772 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6435f44e-980e-4166-868b-ff94fa2010b9-istio-podinfo\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.394923 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.394905 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-istio-envoy\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.394976 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.394960 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6435f44e-980e-4166-868b-ff94fa2010b9-istio-token\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.395062 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.395046 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-workload-certs\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.416243 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.416212 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:34.498893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.498137 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-istio-data\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.498893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.498273 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-workload-socket\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.498893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.498301 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6435f44e-980e-4166-868b-ff94fa2010b9-istio-podinfo\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.498893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.498393 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-istio-envoy\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.498893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.498440 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6435f44e-980e-4166-868b-ff94fa2010b9-istio-token\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.498893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.498469 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-workload-certs\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.498893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.498637 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-credential-socket\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.498893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.498665 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6435f44e-980e-4166-868b-ff94fa2010b9-istiod-ca-cert\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.498893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.498703 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzrm2\" (UniqueName: \"kubernetes.io/projected/6435f44e-980e-4166-868b-ff94fa2010b9-kube-api-access-vzrm2\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.499523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.499499 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-istio-data\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.499770 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.499752 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-workload-socket\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.502657 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.502623 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-workload-certs\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.503297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.503272 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6435f44e-980e-4166-868b-ff94fa2010b9-istio-podinfo\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.503710 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.503689 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-credential-socket\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.504501 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.504448 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6435f44e-980e-4166-868b-ff94fa2010b9-istiod-ca-cert\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.521227 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.521193 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-istio-envoy\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.530944 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.530836 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzrm2\" (UniqueName: \"kubernetes.io/projected/6435f44e-980e-4166-868b-ff94fa2010b9-kube-api-access-vzrm2\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.536700 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.536669 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6435f44e-980e-4166-868b-ff94fa2010b9-istio-token\") pod \"e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:34.561134 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.561105 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:34.649466 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:34.649435 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:35.000445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.000410 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8"] Oct 30 12:34:35.016286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.013307 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:34:35.016286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.013418 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:34:35.016286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.013472 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:34:35.028493 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.028357 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db669f7d-1af0-4f6d-9a65-f3295751ceb8" path="/var/lib/kubelet/pods/db669f7d-1af0-4f6d-9a65-f3295751ceb8/volumes" Oct 30 12:34:35.195094 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.195063 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:35.267282 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.267168 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" event={"ID":"6435f44e-980e-4166-868b-ff94fa2010b9","Type":"ContainerStarted","Data":"0b1529c128f54f2748ccf120adb919179f7481237fa77b6e906805a8aadc1ee9"} Oct 30 12:34:35.269336 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.269310 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" event={"ID":"6435f44e-980e-4166-868b-ff94fa2010b9","Type":"ContainerStarted","Data":"03e900e2d373ae0cfaabe51fa4a08a6e55ccd4ccdeb334e6ae0b6bae4e6bad8c"} Oct 30 12:34:35.327045 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.327005 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:35.337875 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.337798 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" podStartSLOduration=1.337781268 podStartE2EDuration="1.337781268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:35.336439729 +0000 UTC m=+2428.446282611" watchObservedRunningTime="2026-10-30 12:34:35.337781268 +0000 UTC m=+2428.447624152" Oct 30 12:34:35.438084 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.438048 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:35.502267 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.502229 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2"] Oct 30 12:34:35.506656 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.506632 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" Oct 30 12:34:35.524602 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.524503 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2"] Oct 30 12:34:35.526706 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.526678 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2675c0\"" Oct 30 12:34:35.526880 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.526859 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2675c0-dockercfg-q9cqw\"" Oct 30 12:34:35.574151 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.574104 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b"] Oct 30 12:34:35.578456 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.578430 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" Oct 30 12:34:35.597575 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.597539 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b"] Oct 30 12:34:35.642274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.635118 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dmvb\" (UniqueName: \"kubernetes.io/projected/fdabf3ad-4ef3-4f8e-9f53-6565dacf505e-kube-api-access-6dmvb\") pod \"payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2\" (UID: \"fdabf3ad-4ef3-4f8e-9f53-6565dacf505e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" Oct 30 12:34:35.642274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.635225 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdabf3ad-4ef3-4f8e-9f53-6565dacf505e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2\" (UID: \"fdabf3ad-4ef3-4f8e-9f53-6565dacf505e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" Oct 30 12:34:35.661386 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.661354 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:35.740153 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.740110 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dmvb\" (UniqueName: \"kubernetes.io/projected/fdabf3ad-4ef3-4f8e-9f53-6565dacf505e-kube-api-access-6dmvb\") pod \"payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2\" (UID: \"fdabf3ad-4ef3-4f8e-9f53-6565dacf505e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" Oct 30 12:34:35.740803 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.740777 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a11193-ed13-452f-b0b1-eaeb60b21465-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b\" (UID: \"68a11193-ed13-452f-b0b1-eaeb60b21465\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" Oct 30 12:34:35.741018 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.740999 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdabf3ad-4ef3-4f8e-9f53-6565dacf505e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2\" (UID: \"fdabf3ad-4ef3-4f8e-9f53-6565dacf505e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" Oct 30 12:34:35.741371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.741352 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrlnm\" (UniqueName: \"kubernetes.io/projected/68a11193-ed13-452f-b0b1-eaeb60b21465-kube-api-access-vrlnm\") pod \"payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b\" (UID: \"68a11193-ed13-452f-b0b1-eaeb60b21465\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" Oct 30 12:34:35.742462 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.742439 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdabf3ad-4ef3-4f8e-9f53-6565dacf505e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2\" (UID: \"fdabf3ad-4ef3-4f8e-9f53-6565dacf505e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" Oct 30 12:34:35.771402 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.771368 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dmvb\" (UniqueName: \"kubernetes.io/projected/fdabf3ad-4ef3-4f8e-9f53-6565dacf505e-kube-api-access-6dmvb\") pod \"payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2\" (UID: \"fdabf3ad-4ef3-4f8e-9f53-6565dacf505e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" Oct 30 12:34:35.860632 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.860526 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" Oct 30 12:34:35.862954 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.861627 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a11193-ed13-452f-b0b1-eaeb60b21465-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b\" (UID: \"68a11193-ed13-452f-b0b1-eaeb60b21465\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" Oct 30 12:34:35.863413 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.863389 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrlnm\" (UniqueName: \"kubernetes.io/projected/68a11193-ed13-452f-b0b1-eaeb60b21465-kube-api-access-vrlnm\") pod \"payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b\" (UID: \"68a11193-ed13-452f-b0b1-eaeb60b21465\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" Oct 30 12:34:35.864013 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.862663 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a11193-ed13-452f-b0b1-eaeb60b21465-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b\" (UID: \"68a11193-ed13-452f-b0b1-eaeb60b21465\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" Oct 30 12:34:35.925282 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:35.921922 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrlnm\" (UniqueName: \"kubernetes.io/projected/68a11193-ed13-452f-b0b1-eaeb60b21465-kube-api-access-vrlnm\") pod \"payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b\" (UID: \"68a11193-ed13-452f-b0b1-eaeb60b21465\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" Oct 30 12:34:36.115333 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:34:36.104609 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:34:36.138273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:36.137143 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:36.197810 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:36.197507 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" Oct 30 12:34:36.518233 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:36.489375 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:36.621277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:36.620341 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2"] Oct 30 12:34:36.681624 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:36.660871 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:36.725304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:36.722538 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:36.896278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:36.840415 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:36.896278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:36.850532 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:34:36.903847 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:36.903779 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:37.160400 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:37.160353 4413 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:37.186273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:37.161530 4413 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded" Oct 30 12:34:37.336276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:37.325601 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 12:34:37.354901 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:37.354859 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 12:34:37.441461 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:37.422365 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:37.786337 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:37.782381 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:37.851412 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:37.839692 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 12:34:37.851412 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:37.841199 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:37.945311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:37.870443 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" podUID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:38.422711 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:34:38.317890 4413 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.282s" Oct 30 12:34:38.422711 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:38.342431 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b"] Oct 30 12:34:38.422711 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:38.358329 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:38.422711 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:38.359025 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:34:38.487287 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:38.484391 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:38.581403 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:38.581371 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:34:38.582922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:38.582889 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 12:34:38.583504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:38.583484 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:34:38.644359 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:38.613397 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" event={"ID":"fdabf3ad-4ef3-4f8e-9f53-6565dacf505e","Type":"ContainerStarted","Data":"c1df53b5c3bc110edf5d0f388d9ed7426f138e96c0130feee6d22c53d1310303"} Oct 30 12:34:38.644359 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:38.636011 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 12:34:38.700350 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:38.700324 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 12:34:38.700740 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:38.700723 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 12:34:38.967613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:38.967580 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:39.127659 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.127634 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl"] Oct 30 12:34:39.127829 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.127755 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:39.131821 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.131804 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" Oct 30 12:34:39.149599 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.149580 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-812990\"" Oct 30 12:34:39.150325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.150309 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-812990-dockercfg-jdvwr\"" Oct 30 12:34:39.158600 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.158566 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl"] Oct 30 12:34:39.175128 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.175105 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twf4d\" (UniqueName: \"kubernetes.io/projected/e2971b88-5b87-49db-b7f8-60c2a9d7116d-kube-api-access-twf4d\") pod \"payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl\" (UID: \"e2971b88-5b87-49db-b7f8-60c2a9d7116d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" Oct 30 12:34:39.175241 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.175154 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2971b88-5b87-49db-b7f8-60c2a9d7116d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl\" (UID: \"e2971b88-5b87-49db-b7f8-60c2a9d7116d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" Oct 30 12:34:39.185277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.185229 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj"] Oct 30 12:34:39.189394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.189377 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" Oct 30 12:34:39.208634 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.208608 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj"] Oct 30 12:34:39.276447 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.276415 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0818b575-9f29-4241-967b-448947aaf982-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj\" (UID: \"0818b575-9f29-4241-967b-448947aaf982\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" Oct 30 12:34:39.276613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.276487 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twf4d\" (UniqueName: \"kubernetes.io/projected/e2971b88-5b87-49db-b7f8-60c2a9d7116d-kube-api-access-twf4d\") pod \"payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl\" (UID: \"e2971b88-5b87-49db-b7f8-60c2a9d7116d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" Oct 30 12:34:39.276613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.276591 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2971b88-5b87-49db-b7f8-60c2a9d7116d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl\" (UID: \"e2971b88-5b87-49db-b7f8-60c2a9d7116d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" Oct 30 12:34:39.276722 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.276709 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p875z\" (UniqueName: \"kubernetes.io/projected/0818b575-9f29-4241-967b-448947aaf982-kube-api-access-p875z\") pod \"payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj\" (UID: \"0818b575-9f29-4241-967b-448947aaf982\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" Oct 30 12:34:39.277263 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.277234 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2971b88-5b87-49db-b7f8-60c2a9d7116d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl\" (UID: \"e2971b88-5b87-49db-b7f8-60c2a9d7116d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" Oct 30 12:34:39.304164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.304132 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twf4d\" (UniqueName: \"kubernetes.io/projected/e2971b88-5b87-49db-b7f8-60c2a9d7116d-kube-api-access-twf4d\") pod \"payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl\" (UID: \"e2971b88-5b87-49db-b7f8-60c2a9d7116d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" Oct 30 12:34:39.377766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.377683 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p875z\" (UniqueName: \"kubernetes.io/projected/0818b575-9f29-4241-967b-448947aaf982-kube-api-access-p875z\") pod \"payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj\" (UID: \"0818b575-9f29-4241-967b-448947aaf982\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" Oct 30 12:34:39.377766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.377756 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0818b575-9f29-4241-967b-448947aaf982-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj\" (UID: \"0818b575-9f29-4241-967b-448947aaf982\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" Oct 30 12:34:39.378487 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.378469 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0818b575-9f29-4241-967b-448947aaf982-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj\" (UID: \"0818b575-9f29-4241-967b-448947aaf982\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" Oct 30 12:34:39.405118 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.405089 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p875z\" (UniqueName: \"kubernetes.io/projected/0818b575-9f29-4241-967b-448947aaf982-kube-api-access-p875z\") pod \"payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj\" (UID: \"0818b575-9f29-4241-967b-448947aaf982\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" Oct 30 12:34:39.412132 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.412095 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:39.445870 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.445842 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" Oct 30 12:34:39.505669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.505530 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" Oct 30 12:34:39.646370 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.646285 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" event={"ID":"68a11193-ed13-452f-b0b1-eaeb60b21465","Type":"ContainerStarted","Data":"46bc5abd3a94b3010a0f89c1cf734cadbd3fe16580474ee674b503025f9e3226"} Oct 30 12:34:39.646515 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.646394 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" event={"ID":"68a11193-ed13-452f-b0b1-eaeb60b21465","Type":"ContainerStarted","Data":"b3d8a53970c706207082b1354b55cc50fb8cb9dd5671bfdbc884e3a50f53b863"} Oct 30 12:34:39.646705 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.646688 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" Oct 30 12:34:39.653350 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.653313 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" event={"ID":"fdabf3ad-4ef3-4f8e-9f53-6565dacf505e","Type":"ContainerStarted","Data":"baccd1ee02487a3f1a999ea59dfe371f52d2dfa486530e92d16cd2d3fd6f869d"} Oct 30 12:34:39.657380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.657352 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" Oct 30 12:34:39.694285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.694208 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" podStartSLOduration=4.694194017 podStartE2EDuration="4.694194017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:39.693198784 +0000 UTC m=+2432.803041664" watchObservedRunningTime="2026-10-30 12:34:39.694194017 +0000 UTC m=+2432.804036895" Oct 30 12:34:39.741415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.741346 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" podStartSLOduration=4.741327547 podStartE2EDuration="4.741327547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:39.740695038 +0000 UTC m=+2432.850537939" watchObservedRunningTime="2026-10-30 12:34:39.741327547 +0000 UTC m=+2432.851170429" Oct 30 12:34:39.759599 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.759559 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl"] Oct 30 12:34:39.761300 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:34:39.761267 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2971b88_5b87_49db_b7f8_60c2a9d7116d.slice/crio-54d54e6bd74c1c53f08c54aa134985156909cdee0de22c5ee128b87ebbbebe30 WatchSource:0}: Error finding container 54d54e6bd74c1c53f08c54aa134985156909cdee0de22c5ee128b87ebbbebe30: Status 404 returned error can't find the container with id 54d54e6bd74c1c53f08c54aa134985156909cdee0de22c5ee128b87ebbbebe30 Oct 30 12:34:39.832172 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.832140 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:39.923953 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.923921 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj"] Oct 30 12:34:39.926336 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:34:39.926285 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0818b575_9f29_4241_967b_448947aaf982.slice/crio-a906e8d35bf20c1ef635424a681772beb876cb0286775417c9d301e503122d70 WatchSource:0}: Error finding container a906e8d35bf20c1ef635424a681772beb876cb0286775417c9d301e503122d70: Status 404 returned error can't find the container with id a906e8d35bf20c1ef635424a681772beb876cb0286775417c9d301e503122d70 Oct 30 12:34:39.960291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:39.960244 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:40.659666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:40.659632 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" event={"ID":"0818b575-9f29-4241-967b-448947aaf982","Type":"ContainerStarted","Data":"a58fa4287c50b9b20a822260f3e9dd75c85630bc58d1298d813a70f9ba900471"} Oct 30 12:34:40.660159 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:40.659693 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" event={"ID":"0818b575-9f29-4241-967b-448947aaf982","Type":"ContainerStarted","Data":"a906e8d35bf20c1ef635424a681772beb876cb0286775417c9d301e503122d70"} Oct 30 12:34:40.660159 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:40.659838 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" Oct 30 12:34:40.661176 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:40.661147 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" event={"ID":"e2971b88-5b87-49db-b7f8-60c2a9d7116d","Type":"ContainerStarted","Data":"27f9a0f03583d6206ee5731025fcc540132eb075f406148a0b57c4860f39036d"} Oct 30 12:34:40.661323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:40.661190 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" event={"ID":"e2971b88-5b87-49db-b7f8-60c2a9d7116d","Type":"ContainerStarted","Data":"54d54e6bd74c1c53f08c54aa134985156909cdee0de22c5ee128b87ebbbebe30"} Oct 30 12:34:40.701071 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:40.701023 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" podStartSLOduration=1.701010246 podStartE2EDuration="1.701010246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:40.698827849 +0000 UTC m=+2433.808670730" watchObservedRunningTime="2026-10-30 12:34:40.701010246 +0000 UTC m=+2433.810853126" Oct 30 12:34:40.740750 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:40.740700 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" podStartSLOduration=1.740686019 podStartE2EDuration="1.740686019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:40.738519293 +0000 UTC m=+2433.848362175" watchObservedRunningTime="2026-10-30 12:34:40.740686019 +0000 UTC m=+2433.850528899" Oct 30 12:34:41.462015 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:41.461984 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:41.669931 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:41.669897 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" Oct 30 12:34:41.707384 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:41.707352 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:41.816956 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:41.816879 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7"] Oct 30 12:34:41.821211 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:41.821196 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" Oct 30 12:34:41.823754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:41.823723 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:41.859467 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:41.859443 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-0bda67\"" Oct 30 12:34:41.863956 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:41.863928 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7"] Oct 30 12:34:41.885614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:41.885582 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk"] Oct 30 12:34:41.889728 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:41.889704 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" Oct 30 12:34:41.907420 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:41.907399 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-0bda67-dockercfg-xhpm9\"" Oct 30 12:34:41.907974 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:41.907943 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk"] Oct 30 12:34:42.000312 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:41.997827 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2232e7b3-95ec-4866-81a9-30a396bf9d96-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7\" (UID: \"2232e7b3-95ec-4866-81a9-30a396bf9d96\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" Oct 30 12:34:42.000312 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:41.997911 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6qf4\" (UniqueName: \"kubernetes.io/projected/2232e7b3-95ec-4866-81a9-30a396bf9d96-kube-api-access-n6qf4\") pod \"payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7\" (UID: \"2232e7b3-95ec-4866-81a9-30a396bf9d96\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" Oct 30 12:34:42.000312 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:41.997972 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brmgp\" (UniqueName: \"kubernetes.io/projected/272a58bb-d7d5-4c02-a278-1dd57cdc75bf-kube-api-access-brmgp\") pod \"payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk\" (UID: \"272a58bb-d7d5-4c02-a278-1dd57cdc75bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" Oct 30 12:34:42.000312 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:41.998057 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272a58bb-d7d5-4c02-a278-1dd57cdc75bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk\" (UID: \"272a58bb-d7d5-4c02-a278-1dd57cdc75bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" Oct 30 12:34:42.099028 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.098953 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2232e7b3-95ec-4866-81a9-30a396bf9d96-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7\" (UID: \"2232e7b3-95ec-4866-81a9-30a396bf9d96\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" Oct 30 12:34:42.099028 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.099017 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6qf4\" (UniqueName: \"kubernetes.io/projected/2232e7b3-95ec-4866-81a9-30a396bf9d96-kube-api-access-n6qf4\") pod \"payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7\" (UID: \"2232e7b3-95ec-4866-81a9-30a396bf9d96\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" Oct 30 12:34:42.099214 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.099100 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brmgp\" (UniqueName: \"kubernetes.io/projected/272a58bb-d7d5-4c02-a278-1dd57cdc75bf-kube-api-access-brmgp\") pod \"payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk\" (UID: \"272a58bb-d7d5-4c02-a278-1dd57cdc75bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" Oct 30 12:34:42.099214 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.099193 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272a58bb-d7d5-4c02-a278-1dd57cdc75bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk\" (UID: \"272a58bb-d7d5-4c02-a278-1dd57cdc75bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" Oct 30 12:34:42.099876 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.099848 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2232e7b3-95ec-4866-81a9-30a396bf9d96-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7\" (UID: \"2232e7b3-95ec-4866-81a9-30a396bf9d96\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" Oct 30 12:34:42.099983 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.099949 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272a58bb-d7d5-4c02-a278-1dd57cdc75bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk\" (UID: \"272a58bb-d7d5-4c02-a278-1dd57cdc75bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" Oct 30 12:34:42.133780 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.133755 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6qf4\" (UniqueName: \"kubernetes.io/projected/2232e7b3-95ec-4866-81a9-30a396bf9d96-kube-api-access-n6qf4\") pod \"payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7\" (UID: \"2232e7b3-95ec-4866-81a9-30a396bf9d96\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" Oct 30 12:34:42.134196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.134174 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brmgp\" (UniqueName: \"kubernetes.io/projected/272a58bb-d7d5-4c02-a278-1dd57cdc75bf-kube-api-access-brmgp\") pod \"payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk\" (UID: \"272a58bb-d7d5-4c02-a278-1dd57cdc75bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" Oct 30 12:34:42.161760 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.161734 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" Oct 30 12:34:42.204205 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.203595 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" Oct 30 12:34:42.461582 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.461554 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7"] Oct 30 12:34:42.462899 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:34:42.462877 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2232e7b3_95ec_4866_81a9_30a396bf9d96.slice/crio-07e1c44fc4c11e4766903b16c392ce54be1078f4871b77b2c62cb5ee39e71a7d WatchSource:0}: Error finding container 07e1c44fc4c11e4766903b16c392ce54be1078f4871b77b2c62cb5ee39e71a7d: Status 404 returned error can't find the container with id 07e1c44fc4c11e4766903b16c392ce54be1078f4871b77b2c62cb5ee39e71a7d Oct 30 12:34:42.524335 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:34:42.524301 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272a58bb_d7d5_4c02_a278_1dd57cdc75bf.slice/crio-2c8b125fbb6c3b2fc0058db0ac680cdaa1c10a3c7d175cf061edde00bf108a34 WatchSource:0}: Error finding container 2c8b125fbb6c3b2fc0058db0ac680cdaa1c10a3c7d175cf061edde00bf108a34: Status 404 returned error can't find the container with id 2c8b125fbb6c3b2fc0058db0ac680cdaa1c10a3c7d175cf061edde00bf108a34 Oct 30 12:34:42.525063 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.525039 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk"] Oct 30 12:34:42.581607 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.581581 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29"] Oct 30 12:34:42.581812 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.581788 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" podUID="9bebc04d-dded-466d-932f-0dfd4eec3183" containerName="istio-proxy" containerID="cri-o://7d28111bdfaf3e767dbca32bb00d04889a7c460b236fac01b8a3e4be7a34b2dc" gracePeriod=30 Oct 30 12:34:42.677431 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.677396 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" event={"ID":"2232e7b3-95ec-4866-81a9-30a396bf9d96","Type":"ContainerStarted","Data":"19d4d13dd4df519ef05bd56f60b6da4c13e1499a81ab9406a9ba96c7477ca693"} Oct 30 12:34:42.677859 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.677455 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" event={"ID":"2232e7b3-95ec-4866-81a9-30a396bf9d96","Type":"ContainerStarted","Data":"07e1c44fc4c11e4766903b16c392ce54be1078f4871b77b2c62cb5ee39e71a7d"} Oct 30 12:34:42.677859 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.677591 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" Oct 30 12:34:42.679082 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.679061 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" event={"ID":"272a58bb-d7d5-4c02-a278-1dd57cdc75bf","Type":"ContainerStarted","Data":"b7cdc327b48b65efd776da2fb70673c974754f0b6f198236741be6b58bfd0b65"} Oct 30 12:34:42.679210 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.679098 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" event={"ID":"272a58bb-d7d5-4c02-a278-1dd57cdc75bf","Type":"ContainerStarted","Data":"2c8b125fbb6c3b2fc0058db0ac680cdaa1c10a3c7d175cf061edde00bf108a34"} Oct 30 12:34:42.679297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.679214 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" Oct 30 12:34:42.719950 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.719845 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" podStartSLOduration=1.71983106 podStartE2EDuration="1.71983106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:42.717400488 +0000 UTC m=+2435.827243430" watchObservedRunningTime="2026-10-30 12:34:42.71983106 +0000 UTC m=+2435.829673941" Oct 30 12:34:42.760106 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:42.760051 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" podStartSLOduration=1.7600367160000001 podStartE2EDuration="1.760036716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:42.756433814 +0000 UTC m=+2435.866276719" watchObservedRunningTime="2026-10-30 12:34:42.760036716 +0000 UTC m=+2435.869879599" Oct 30 12:34:43.334449 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:43.334419 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:43.464769 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:43.464733 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:43.688226 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:43.688187 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:45.183565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:45.183534 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:47.276760 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.276708 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:47.766606 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.766512 4413 generic.go:356] "Generic (PLEG): container finished" podID="9bebc04d-dded-466d-932f-0dfd4eec3183" containerID="7d28111bdfaf3e767dbca32bb00d04889a7c460b236fac01b8a3e4be7a34b2dc" exitCode=0 Oct 30 12:34:47.766606 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.766588 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" event={"ID":"9bebc04d-dded-466d-932f-0dfd4eec3183","Type":"ContainerDied","Data":"7d28111bdfaf3e767dbca32bb00d04889a7c460b236fac01b8a3e4be7a34b2dc"} Oct 30 12:34:47.886402 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.886375 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:34:47.986381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.986353 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-workload-certs\") pod \"9bebc04d-dded-466d-932f-0dfd4eec3183\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " Oct 30 12:34:47.986595 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.986392 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-workload-socket\") pod \"9bebc04d-dded-466d-932f-0dfd4eec3183\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " Oct 30 12:34:47.986595 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.986443 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-envoy\") pod \"9bebc04d-dded-466d-932f-0dfd4eec3183\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " Oct 30 12:34:47.986595 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.986539 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bebc04d-dded-466d-932f-0dfd4eec3183-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bebc04d-dded-466d-932f-0dfd4eec3183-istiod-ca-cert\") pod \"9bebc04d-dded-466d-932f-0dfd4eec3183\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " Oct 30 12:34:47.986777 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.986654 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-podinfo\") pod \"9bebc04d-dded-466d-932f-0dfd4eec3183\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " Oct 30 12:34:47.986777 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.986684 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-credential-socket\") pod \"9bebc04d-dded-466d-932f-0dfd4eec3183\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " Oct 30 12:34:47.986777 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.986710 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-workload-socket" pod "9bebc04d-dded-466d-932f-0dfd4eec3183" (UID: "9bebc04d-dded-466d-932f-0dfd4eec3183"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:47.986777 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.986746 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-data\") pod \"9bebc04d-dded-466d-932f-0dfd4eec3183\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " Oct 30 12:34:47.986991 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.986786 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-workload-certs" pod "9bebc04d-dded-466d-932f-0dfd4eec3183" (UID: "9bebc04d-dded-466d-932f-0dfd4eec3183"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:47.986991 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.986948 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-data" pod "9bebc04d-dded-466d-932f-0dfd4eec3183" (UID: "9bebc04d-dded-466d-932f-0dfd4eec3183"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:47.987098 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.987034 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-credential-socket" pod "9bebc04d-dded-466d-932f-0dfd4eec3183" (UID: "9bebc04d-dded-466d-932f-0dfd4eec3183"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:47.987098 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.987078 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bebc04d-dded-466d-932f-0dfd4eec3183-istiod-ca-cert" pod "9bebc04d-dded-466d-932f-0dfd4eec3183" (UID: "9bebc04d-dded-466d-932f-0dfd4eec3183"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:34:47.987211 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.987117 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bebc04d-dded-466d-932f-0dfd4eec3183-kube-api-access-mjrjg\" (UniqueName: \"kubernetes.io/projected/9bebc04d-dded-466d-932f-0dfd4eec3183-kube-api-access-mjrjg\") pod \"9bebc04d-dded-466d-932f-0dfd4eec3183\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " Oct 30 12:34:47.987211 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.987144 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-token\" (UniqueName: \"kubernetes.io/projected/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-token\") pod \"9bebc04d-dded-466d-932f-0dfd4eec3183\" (UID: \"9bebc04d-dded-466d-932f-0dfd4eec3183\") " Oct 30 12:34:47.988197 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.987802 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:47.988197 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.987829 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:47.988197 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.987846 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bebc04d-dded-466d-932f-0dfd4eec3183-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:47.988197 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.987860 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:47.988197 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.987923 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:47.989900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.989817 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-envoy" pod "9bebc04d-dded-466d-932f-0dfd4eec3183" (UID: "9bebc04d-dded-466d-932f-0dfd4eec3183"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:34:47.990013 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.989982 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-podinfo" pod "9bebc04d-dded-466d-932f-0dfd4eec3183" (UID: "9bebc04d-dded-466d-932f-0dfd4eec3183"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:34:47.990227 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.990202 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bebc04d-dded-466d-932f-0dfd4eec3183-kube-api-access-mjrjg" pod "9bebc04d-dded-466d-932f-0dfd4eec3183" (UID: "9bebc04d-dded-466d-932f-0dfd4eec3183"). InnerVolumeSpecName "kube-api-access-mjrjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:47.990349 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:47.990326 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-token" pod "9bebc04d-dded-466d-932f-0dfd4eec3183" (UID: "9bebc04d-dded-466d-932f-0dfd4eec3183"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:34:48.089811 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:48.089785 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:48.089811 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:48.089808 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjrjg\" (UniqueName: \"kubernetes.io/projected/9bebc04d-dded-466d-932f-0dfd4eec3183-kube-api-access-mjrjg\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:48.089811 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:48.089818 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:48.089811 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:48.089826 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bebc04d-dded-466d-932f-0dfd4eec3183-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:34:48.774904 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:48.774870 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" Oct 30 12:34:48.774904 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:48.774878 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29" event={"ID":"9bebc04d-dded-466d-932f-0dfd4eec3183","Type":"ContainerDied","Data":"76abd6c8693403b0c6bee1f5700eeba945098e507b12e15c62570591922aabe7"} Oct 30 12:34:48.775504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:48.774953 4413 scope.go:122] "RemoveContainer" containerID="7d28111bdfaf3e767dbca32bb00d04889a7c460b236fac01b8a3e4be7a34b2dc" Oct 30 12:34:48.838316 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:48.838284 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29"] Oct 30 12:34:48.849855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:48.849828 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-h7h29"] Oct 30 12:34:48.932463 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:48.932433 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs"] Oct 30 12:34:48.933023 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:48.933009 4413 cpu_manager.go:414] "containerMap: removing container" podUID="9bebc04d-dded-466d-932f-0dfd4eec3183" containerName="istio-proxy" Oct 30 12:34:48.933061 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:48.933025 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bebc04d-dded-466d-932f-0dfd4eec3183" containerName="istio-proxy" Oct 30 12:34:48.933124 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:48.933116 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bebc04d-dded-466d-932f-0dfd4eec3183" containerName="istio-proxy" Oct 30 12:34:48.939372 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:48.939348 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" Oct 30 12:34:48.957510 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:48.957483 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-2675c0-serving-cert\"" Oct 30 12:34:48.957839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:48.957799 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs"] Oct 30 12:34:49.022317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.022282 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bebc04d-dded-466d-932f-0dfd4eec3183" path="/var/lib/kubelet/pods/9bebc04d-dded-466d-932f-0dfd4eec3183/volumes" Oct 30 12:34:49.028776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.028718 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:49.102844 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.102813 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94ea46b-0895-4536-a9c6-4088e1116400-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs\" (UID: \"d94ea46b-0895-4536-a9c6-4088e1116400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" Oct 30 12:34:49.103069 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.102959 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d94ea46b-0895-4536-a9c6-4088e1116400-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs\" (UID: \"d94ea46b-0895-4536-a9c6-4088e1116400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" Oct 30 12:34:49.103069 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.102993 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d94ea46b-0895-4536-a9c6-4088e1116400-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs\" (UID: \"d94ea46b-0895-4536-a9c6-4088e1116400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" Oct 30 12:34:49.103069 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.103042 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mmpf\" (UniqueName: \"kubernetes.io/projected/d94ea46b-0895-4536-a9c6-4088e1116400-kube-api-access-2mmpf\") pod \"maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs\" (UID: \"d94ea46b-0895-4536-a9c6-4088e1116400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" Oct 30 12:34:49.138479 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.138448 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:49.204034 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.204005 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94ea46b-0895-4536-a9c6-4088e1116400-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs\" (UID: \"d94ea46b-0895-4536-a9c6-4088e1116400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" Oct 30 12:34:49.204241 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.204126 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d94ea46b-0895-4536-a9c6-4088e1116400-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs\" (UID: \"d94ea46b-0895-4536-a9c6-4088e1116400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" Oct 30 12:34:49.204241 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.204157 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d94ea46b-0895-4536-a9c6-4088e1116400-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs\" (UID: \"d94ea46b-0895-4536-a9c6-4088e1116400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" Oct 30 12:34:49.204423 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.204351 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mmpf\" (UniqueName: \"kubernetes.io/projected/d94ea46b-0895-4536-a9c6-4088e1116400-kube-api-access-2mmpf\") pod \"maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs\" (UID: \"d94ea46b-0895-4536-a9c6-4088e1116400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" Oct 30 12:34:49.204872 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.204844 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94ea46b-0895-4536-a9c6-4088e1116400-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs\" (UID: \"d94ea46b-0895-4536-a9c6-4088e1116400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" Oct 30 12:34:49.206907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.206887 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d94ea46b-0895-4536-a9c6-4088e1116400-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs\" (UID: \"d94ea46b-0895-4536-a9c6-4088e1116400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" Oct 30 12:34:49.207014 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.206931 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d94ea46b-0895-4536-a9c6-4088e1116400-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs\" (UID: \"d94ea46b-0895-4536-a9c6-4088e1116400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" Oct 30 12:34:49.231978 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.231949 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mmpf\" (UniqueName: \"kubernetes.io/projected/d94ea46b-0895-4536-a9c6-4088e1116400-kube-api-access-2mmpf\") pod \"maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs\" (UID: \"d94ea46b-0895-4536-a9c6-4088e1116400\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" Oct 30 12:34:49.255607 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.255575 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" Oct 30 12:34:49.264755 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.264725 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:49.563517 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.563488 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs"] Oct 30 12:34:49.566047 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:34:49.565975 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94ea46b_0895_4536_a9c6_4088e1116400.slice/crio-bb4ed41ce38e01cf2193ea9cea119659d736411d1782b9800c018be7b73bc4db WatchSource:0}: Error finding container bb4ed41ce38e01cf2193ea9cea119659d736411d1782b9800c018be7b73bc4db: Status 404 returned error can't find the container with id bb4ed41ce38e01cf2193ea9cea119659d736411d1782b9800c018be7b73bc4db Oct 30 12:34:49.784160 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:49.784127 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" event={"ID":"d94ea46b-0895-4536-a9c6-4088e1116400","Type":"ContainerStarted","Data":"bb4ed41ce38e01cf2193ea9cea119659d736411d1782b9800c018be7b73bc4db"} Oct 30 12:34:50.664844 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.664818 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" Oct 30 12:34:50.725022 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.724983 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn"] Oct 30 12:34:50.730262 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.730216 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" Oct 30 12:34:50.750441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.750409 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn"] Oct 30 12:34:50.750609 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.750477 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-812990-serving-cert\"" Oct 30 12:34:50.805891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.805854 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" event={"ID":"d94ea46b-0895-4536-a9c6-4088e1116400","Type":"ContainerStarted","Data":"28ae203f5d4784887f343bf9d9d2ea4abc3cab289b2301cdb6351d7ce03ad03e"} Oct 30 12:34:50.806309 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.805949 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" Oct 30 12:34:50.819592 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.819556 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:50.821883 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.821857 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bfd491e-874f-4edd-b01b-b5bc8d20278b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn\" (UID: \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" Oct 30 12:34:50.822061 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.822039 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bfd491e-874f-4edd-b01b-b5bc8d20278b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn\" (UID: \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" Oct 30 12:34:50.822110 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.822092 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bfd491e-874f-4edd-b01b-b5bc8d20278b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn\" (UID: \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" Oct 30 12:34:50.822429 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.822406 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgrjj\" (UniqueName: \"kubernetes.io/projected/5bfd491e-874f-4edd-b01b-b5bc8d20278b-kube-api-access-jgrjj\") pod \"maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn\" (UID: \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" Oct 30 12:34:50.846936 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.846878 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" podStartSLOduration=2.392829215 podStartE2EDuration="2.846864444s" totalImagesPullingTime="454.035229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:50.846066346 +0000 UTC m=+2443.955909227" watchObservedRunningTime="2026-10-30 12:34:50.846864444 +0000 UTC m=+2443.956707366" Oct 30 12:34:50.923456 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.923371 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bfd491e-874f-4edd-b01b-b5bc8d20278b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn\" (UID: \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" Oct 30 12:34:50.923456 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.923416 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bfd491e-874f-4edd-b01b-b5bc8d20278b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn\" (UID: \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" Oct 30 12:34:50.923713 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.923501 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgrjj\" (UniqueName: \"kubernetes.io/projected/5bfd491e-874f-4edd-b01b-b5bc8d20278b-kube-api-access-jgrjj\") pod \"maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn\" (UID: \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" Oct 30 12:34:50.923713 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.923538 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bfd491e-874f-4edd-b01b-b5bc8d20278b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn\" (UID: \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" Oct 30 12:34:50.924152 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.924127 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bfd491e-874f-4edd-b01b-b5bc8d20278b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn\" (UID: \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" Oct 30 12:34:50.926116 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.926091 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bfd491e-874f-4edd-b01b-b5bc8d20278b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn\" (UID: \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" Oct 30 12:34:50.926207 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.926153 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bfd491e-874f-4edd-b01b-b5bc8d20278b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn\" (UID: \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" Oct 30 12:34:50.930297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.930276 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:50.949926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:50.949856 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgrjj\" (UniqueName: \"kubernetes.io/projected/5bfd491e-874f-4edd-b01b-b5bc8d20278b-kube-api-access-jgrjj\") pod \"maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn\" (UID: \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" Oct 30 12:34:51.050411 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:51.050382 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" Oct 30 12:34:51.311839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:51.311769 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:51.355185 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:51.355157 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn"] Oct 30 12:34:51.356425 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:34:51.356401 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfd491e_874f_4edd_b01b_b5bc8d20278b.slice/crio-71b27408ce37910252991bee6078a6e9780ea5504b42740246ac76241a098081 WatchSource:0}: Error finding container 71b27408ce37910252991bee6078a6e9780ea5504b42740246ac76241a098081: Status 404 returned error can't find the container with id 71b27408ce37910252991bee6078a6e9780ea5504b42740246ac76241a098081 Oct 30 12:34:51.672599 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:51.672571 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" Oct 30 12:34:51.672784 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:51.672764 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" Oct 30 12:34:51.803054 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:51.803018 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:51.815473 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:51.815438 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" event={"ID":"5bfd491e-874f-4edd-b01b-b5bc8d20278b","Type":"ContainerStarted","Data":"a2500660e94b67ad0df3124833da41e9cdaa7d4cb9925d267b737a85739f8793"} Oct 30 12:34:51.815940 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:51.815494 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" event={"ID":"5bfd491e-874f-4edd-b01b-b5bc8d20278b","Type":"ContainerStarted","Data":"71b27408ce37910252991bee6078a6e9780ea5504b42740246ac76241a098081"} Oct 30 12:34:51.815940 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:51.815523 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" Oct 30 12:34:51.857893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:51.857836 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" podStartSLOduration=1.534032956 podStartE2EDuration="1.857813825s" totalImagesPullingTime="323.780869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:51.856380424 +0000 UTC m=+2444.966223318" watchObservedRunningTime="2026-10-30 12:34:51.857813825 +0000 UTC m=+2444.967656705" Oct 30 12:34:51.982864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:51.982768 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:52.651124 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:52.651079 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:52.683130 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:52.683100 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" Oct 30 12:34:53.692006 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:53.691979 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" Oct 30 12:34:53.692553 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:53.692063 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" Oct 30 12:34:54.352672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:54.352639 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:55.753505 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:55.753472 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:55.867873 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:55.867846 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:56.189484 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:56.189456 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:56.330798 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:56.330766 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:56.821323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:56.821295 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" Oct 30 12:34:57.708017 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.707990 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:57.768741 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.768709 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489"] Oct 30 12:34:57.773232 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.773206 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.807683 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.807656 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489"] Oct 30 12:34:57.823195 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.823163 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.823907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.823218 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.823907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.823375 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.823907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.823434 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.823907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.823555 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.823907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.823661 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.823907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.823888 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl88q\" (UniqueName: \"kubernetes.io/projected/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-kube-api-access-zl88q\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.824365 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.823988 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.824365 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.824085 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.832755 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.832719 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" Oct 30 12:34:57.834453 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.834426 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq"] Oct 30 12:34:57.838697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.838678 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" Oct 30 12:34:57.857069 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.857047 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq"] Oct 30 12:34:57.859575 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.859550 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-327e31\"" Oct 30 12:34:57.864779 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.864753 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:57.867058 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.867031 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh"] Oct 30 12:34:57.872342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.872324 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" Oct 30 12:34:57.898016 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.897973 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-327e31-dockercfg-g8tmt\"" Oct 30 12:34:57.899150 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.898911 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh"] Oct 30 12:34:57.926065 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.926030 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.926291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.926109 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.926291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.926160 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.926291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.926207 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.926487 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.926300 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.926487 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.926371 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl88q\" (UniqueName: \"kubernetes.io/projected/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-kube-api-access-zl88q\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.926487 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.926447 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.926646 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.926483 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp4d9\" (UniqueName: \"kubernetes.io/projected/ad21c9c0-28e9-4e5c-b82b-64747d88bc93-kube-api-access-pp4d9\") pod \"payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh\" (UID: \"ad21c9c0-28e9-4e5c-b82b-64747d88bc93\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" Oct 30 12:34:57.926646 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.926567 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.926646 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.926626 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad21c9c0-28e9-4e5c-b82b-64747d88bc93-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh\" (UID: \"ad21c9c0-28e9-4e5c-b82b-64747d88bc93\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" Oct 30 12:34:57.926806 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.926657 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25322f93-360d-472f-8c1b-382c04505663-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq\" (UID: \"25322f93-360d-472f-8c1b-382c04505663\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" Oct 30 12:34:57.926806 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.926696 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9qxc\" (UniqueName: \"kubernetes.io/projected/25322f93-360d-472f-8c1b-382c04505663-kube-api-access-l9qxc\") pod \"payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq\" (UID: \"25322f93-360d-472f-8c1b-382c04505663\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" Oct 30 12:34:57.926806 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.926781 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.927493 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.927469 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istiod-ca-cert\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.927906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.927887 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-workload-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.928396 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.928370 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-data\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.928499 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.928465 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-workload-certs\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.928746 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.928720 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-credential-socket\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.931000 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.930978 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-envoy\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.931342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.931323 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-podinfo\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.949891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.949866 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl88q\" (UniqueName: \"kubernetes.io/projected/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-kube-api-access-zl88q\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:57.950292 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:57.950270 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-token\") pod \"e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:58.027684 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.027605 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad21c9c0-28e9-4e5c-b82b-64747d88bc93-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh\" (UID: \"ad21c9c0-28e9-4e5c-b82b-64747d88bc93\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" Oct 30 12:34:58.027684 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.027636 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25322f93-360d-472f-8c1b-382c04505663-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq\" (UID: \"25322f93-360d-472f-8c1b-382c04505663\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" Oct 30 12:34:58.027684 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.027661 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9qxc\" (UniqueName: \"kubernetes.io/projected/25322f93-360d-472f-8c1b-382c04505663-kube-api-access-l9qxc\") pod \"payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq\" (UID: \"25322f93-360d-472f-8c1b-382c04505663\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" Oct 30 12:34:58.027995 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.027812 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp4d9\" (UniqueName: \"kubernetes.io/projected/ad21c9c0-28e9-4e5c-b82b-64747d88bc93-kube-api-access-pp4d9\") pod \"payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh\" (UID: \"ad21c9c0-28e9-4e5c-b82b-64747d88bc93\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" Oct 30 12:34:58.028371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.028347 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad21c9c0-28e9-4e5c-b82b-64747d88bc93-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh\" (UID: \"ad21c9c0-28e9-4e5c-b82b-64747d88bc93\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" Oct 30 12:34:58.028464 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.028415 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25322f93-360d-472f-8c1b-382c04505663-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq\" (UID: \"25322f93-360d-472f-8c1b-382c04505663\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" Oct 30 12:34:58.050642 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.050619 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp4d9\" (UniqueName: \"kubernetes.io/projected/ad21c9c0-28e9-4e5c-b82b-64747d88bc93-kube-api-access-pp4d9\") pod \"payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh\" (UID: \"ad21c9c0-28e9-4e5c-b82b-64747d88bc93\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" Oct 30 12:34:58.050742 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.050688 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9qxc\" (UniqueName: \"kubernetes.io/projected/25322f93-360d-472f-8c1b-382c04505663-kube-api-access-l9qxc\") pod \"payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq\" (UID: \"25322f93-360d-472f-8c1b-382c04505663\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" Oct 30 12:34:58.109274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.095535 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:58.181270 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.178664 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" Oct 30 12:34:58.197151 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.188912 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" Oct 30 12:34:58.232419 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.229502 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:58.505634 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.505606 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489"] Oct 30 12:34:58.516982 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:34:58.516932 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4847ddfc_5276_42d7_aa4e_33811d0c0dbe.slice/crio-bb593adfa34228d27ce56b5d562cd963b1539f9a1df6e18e0d688fd965d9ee05 WatchSource:0}: Error finding container bb593adfa34228d27ce56b5d562cd963b1539f9a1df6e18e0d688fd965d9ee05: Status 404 returned error can't find the container with id bb593adfa34228d27ce56b5d562cd963b1539f9a1df6e18e0d688fd965d9ee05 Oct 30 12:34:58.519761 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.519722 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:34:58.519852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.519795 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:34:58.519852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.519832 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:34:58.545865 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.543299 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq"] Oct 30 12:34:58.545865 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:34:58.543784 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25322f93_360d_472f_8c1b_382c04505663.slice/crio-5e1c65e3176f0d2d088fd7a8c845e7ef623325ef7021c76e93e7219d8f623bf0 WatchSource:0}: Error finding container 5e1c65e3176f0d2d088fd7a8c845e7ef623325ef7021c76e93e7219d8f623bf0: Status 404 returned error can't find the container with id 5e1c65e3176f0d2d088fd7a8c845e7ef623325ef7021c76e93e7219d8f623bf0 Oct 30 12:34:58.547057 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.547035 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh"] Oct 30 12:34:58.551402 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:34:58.551377 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad21c9c0_28e9_4e5c_b82b_64747d88bc93.slice/crio-2db54fb79f6ec91f07a016251c2209e127b9c34cd165167d2a670823a76fa1c6 WatchSource:0}: Error finding container 2db54fb79f6ec91f07a016251c2209e127b9c34cd165167d2a670823a76fa1c6: Status 404 returned error can't find the container with id 2db54fb79f6ec91f07a016251c2209e127b9c34cd165167d2a670823a76fa1c6 Oct 30 12:34:58.742298 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.736534 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:58.872984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.872893 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" event={"ID":"ad21c9c0-28e9-4e5c-b82b-64747d88bc93","Type":"ContainerStarted","Data":"f32d0b51f39127ed0d7682e98b7bd96ec9cfbfcb19c7de4f94c99906ba2e7250"} Oct 30 12:34:58.872984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.872950 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" event={"ID":"ad21c9c0-28e9-4e5c-b82b-64747d88bc93","Type":"ContainerStarted","Data":"2db54fb79f6ec91f07a016251c2209e127b9c34cd165167d2a670823a76fa1c6"} Oct 30 12:34:58.873497 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.873020 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" Oct 30 12:34:58.874924 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.874895 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" event={"ID":"4847ddfc-5276-42d7-aa4e-33811d0c0dbe","Type":"ContainerStarted","Data":"87f260a4e4ed66f368af0d4eabea1e6c1e760a15c4afba61b2dd73e1f1409886"} Oct 30 12:34:58.875069 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.874941 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" event={"ID":"4847ddfc-5276-42d7-aa4e-33811d0c0dbe","Type":"ContainerStarted","Data":"bb593adfa34228d27ce56b5d562cd963b1539f9a1df6e18e0d688fd965d9ee05"} Oct 30 12:34:58.876558 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.876538 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" event={"ID":"25322f93-360d-472f-8c1b-382c04505663","Type":"ContainerStarted","Data":"88d673a10a442aa4ac5964d6c211158cb2ca45e80b66c3105693d43ec2f9c06b"} Oct 30 12:34:58.876645 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.876574 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" event={"ID":"25322f93-360d-472f-8c1b-382c04505663","Type":"ContainerStarted","Data":"5e1c65e3176f0d2d088fd7a8c845e7ef623325ef7021c76e93e7219d8f623bf0"} Oct 30 12:34:58.876976 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.876961 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" Oct 30 12:34:58.915761 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.915712 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" podStartSLOduration=1.9156988209999999 podStartE2EDuration="1.915698821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:58.912717927 +0000 UTC m=+2452.022560827" watchObservedRunningTime="2026-10-30 12:34:58.915698821 +0000 UTC m=+2452.025541701" Oct 30 12:34:58.924496 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.924438 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:58.963805 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:58.963761 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" podStartSLOduration=1.963748261 podStartE2EDuration="1.963748261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:58.960013269 +0000 UTC m=+2452.069856152" watchObservedRunningTime="2026-10-30 12:34:58.963748261 +0000 UTC m=+2452.073591142" Oct 30 12:34:59.009731 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:59.009675 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" podStartSLOduration=2.009661284 podStartE2EDuration="2.009661284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:34:59.008504414 +0000 UTC m=+2452.118347295" watchObservedRunningTime="2026-10-30 12:34:59.009661284 +0000 UTC m=+2452.119504165" Oct 30 12:34:59.081521 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:59.081487 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:59.099311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:59.099270 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:34:59.100891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:59.100873 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 12:34:59.100955 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:59.100918 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" podUID="4847ddfc-5276-42d7-aa4e-33811d0c0dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 12:34:59.197614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:59.197578 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:34:59.572363 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:34:59.572289 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:00.116276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:00.116186 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 12:35:00.116757 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:00.116242 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" podUID="4847ddfc-5276-42d7-aa4e-33811d0c0dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 12:35:01.110045 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:01.109601 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 12:35:01.110045 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:01.109672 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" podUID="4847ddfc-5276-42d7-aa4e-33811d0c0dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 12:35:01.261405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:01.261378 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:02.114275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:02.113464 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 12:35:02.114275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:02.113525 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" podUID="4847ddfc-5276-42d7-aa4e-33811d0c0dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 12:35:03.078113 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:03.078050 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:03.122273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:03.121560 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 12:35:03.122273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:03.121624 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" podUID="4847ddfc-5276-42d7-aa4e-33811d0c0dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 12:35:04.122275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:04.121765 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 12:35:04.122275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:04.121832 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" podUID="4847ddfc-5276-42d7-aa4e-33811d0c0dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 12:35:04.781900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:04.781858 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:05.099818 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:05.099716 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 12:35:05.100099 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:05.100068 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" podUID="4847ddfc-5276-42d7-aa4e-33811d0c0dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 12:35:05.846661 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:35:05.846621 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:35:06.192274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:06.192018 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:06.313277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:06.312771 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:06.419276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:06.418589 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:35:06.419499 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:06.419285 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:35:06.421653 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:06.421624 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:35:06.566608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:06.566490 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:06.687122 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:06.685180 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:08.280088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:08.279995 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:09.316905 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:09.316868 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:09.801707 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:09.801651 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:09.890907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:09.890873 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" Oct 30 12:35:10.306790 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:10.306736 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:10.429798 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:10.429284 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:10.893044 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:10.893015 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" Oct 30 12:35:12.024138 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:12.024102 4413 generic.go:356] "Generic (PLEG): container finished" podID="75e43f0b-df10-4b4e-b7af-67d515178a84" containerID="e1f4a8c880313b406b4cd2b36e5344c75bdc8e7408a619bb8f538930f572d732" exitCode=0 Oct 30 12:35:12.024612 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:12.024170 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" event={"ID":"75e43f0b-df10-4b4e-b7af-67d515178a84","Type":"ContainerDied","Data":"e1f4a8c880313b406b4cd2b36e5344c75bdc8e7408a619bb8f538930f572d732"} Oct 30 12:35:12.024675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:12.024613 4413 scope.go:122] "RemoveContainer" containerID="e1f4a8c880313b406b4cd2b36e5344c75bdc8e7408a619bb8f538930f572d732" Oct 30 12:35:12.026048 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:12.026020 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:35:12.026113 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:12.026088 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:35:12.026152 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:12.026120 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:35:12.141750 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:12.141668 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:13.032083 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:13.032052 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" event={"ID":"75e43f0b-df10-4b4e-b7af-67d515178a84","Type":"ContainerStarted","Data":"e6f0e6d5056b87faa76e90092b3215e9f9c06fa47b9a12684da57d8840c9501c"} Oct 30 12:35:13.874555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:13.874523 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:13.929281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:13.929217 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:35:13.930650 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:13.930625 4413 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 12:35:13.930778 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:13.930672 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" podUID="75e43f0b-df10-4b4e-b7af-67d515178a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 12:35:14.013748 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:14.012591 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:14.157481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:14.157313 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:14.300993 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:14.300958 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:14.584757 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:14.584673 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:14.928964 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:14.928927 4413 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 12:35:14.929159 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:14.928994 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" podUID="75e43f0b-df10-4b4e-b7af-67d515178a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 12:35:15.076201 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.076165 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:15.529720 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.529676 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q"] Oct 30 12:35:15.544137 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.544109 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q"] Oct 30 12:35:15.544479 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.544425 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.557995 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.557961 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:15.560288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.560240 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-42220c-openshift-default-dockercfg-bg6sq\"" Oct 30 12:35:15.679833 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.679792 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-workload-certs\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.680107 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.680083 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-credential-socket\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.680283 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.680243 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm97j\" (UniqueName: \"kubernetes.io/projected/0e06b3dc-2413-4f9f-ba0d-691519d450f1-kube-api-access-qm97j\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.682042 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.682015 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istiod-ca-cert\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.682352 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.682194 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-token\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.685111 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.685082 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-data\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.685319 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.685301 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-workload-socket\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.685591 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.685556 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-podinfo\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.685873 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.685854 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-envoy\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.787395 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.787309 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istiod-ca-cert\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.787662 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.787640 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-token\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.787801 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.787785 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-data\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.787962 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.787946 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-workload-socket\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.788133 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.787961 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istiod-ca-cert\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.788209 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.788133 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-workload-socket\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.788287 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.788214 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-data\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.788403 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.788385 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-podinfo\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.788603 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.788586 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-envoy\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.788721 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.788708 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-workload-certs\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.788824 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.788812 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-credential-socket\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.789297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.789274 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-credential-socket\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.789445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.789308 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm97j\" (UniqueName: \"kubernetes.io/projected/0e06b3dc-2413-4f9f-ba0d-691519d450f1-kube-api-access-qm97j\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.789730 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.789079 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-workload-certs\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.822277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.821061 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-podinfo\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.824215 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.824175 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-envoy\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.826093 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.825772 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-token\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.826285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.826130 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm97j\" (UniqueName: \"kubernetes.io/projected/0e06b3dc-2413-4f9f-ba0d-691519d450f1-kube-api-access-qm97j\") pod \"e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.864894 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.864857 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:15.928793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.928752 4413 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 12:35:15.928951 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:15.928840 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" podUID="75e43f0b-df10-4b4e-b7af-67d515178a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 12:35:16.040370 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.039350 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:16.152862 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.152821 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:16.223122 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.222722 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q"] Oct 30 12:35:16.225828 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:35:16.225794 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e06b3dc_2413_4f9f_ba0d_691519d450f1.slice/crio-dd51b0c4d34a149d3817b88e27f4f3ffe57f0975c8f0760531c38745c5ecc798 WatchSource:0}: Error finding container dd51b0c4d34a149d3817b88e27f4f3ffe57f0975c8f0760531c38745c5ecc798: Status 404 returned error can't find the container with id dd51b0c4d34a149d3817b88e27f4f3ffe57f0975c8f0760531c38745c5ecc798 Oct 30 12:35:16.228698 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.228661 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:35:16.228805 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.228733 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:35:16.228805 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.228763 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:35:16.229016 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.228996 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:35:16.319285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.316310 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:16.321942 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.321613 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j"] Oct 30 12:35:16.325969 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.325899 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" Oct 30 12:35:16.333066 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.333034 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j"] Oct 30 12:35:16.359270 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.358202 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-9d45fb\"" Oct 30 12:35:16.481278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.359994 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-9d45fb-dockercfg-t7qwg\"" Oct 30 12:35:16.583777 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.583693 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms5tc\" (UniqueName: \"kubernetes.io/projected/9aa641d6-2556-4e48-9503-85d33c43df8a-kube-api-access-ms5tc\") pod \"payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j\" (UID: \"9aa641d6-2556-4e48-9503-85d33c43df8a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" Oct 30 12:35:16.584144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.583811 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa641d6-2556-4e48-9503-85d33c43df8a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j\" (UID: \"9aa641d6-2556-4e48-9503-85d33c43df8a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" Oct 30 12:35:16.685375 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.685339 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa641d6-2556-4e48-9503-85d33c43df8a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j\" (UID: \"9aa641d6-2556-4e48-9503-85d33c43df8a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" Oct 30 12:35:16.685562 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.685465 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms5tc\" (UniqueName: \"kubernetes.io/projected/9aa641d6-2556-4e48-9503-85d33c43df8a-kube-api-access-ms5tc\") pod \"payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j\" (UID: \"9aa641d6-2556-4e48-9503-85d33c43df8a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" Oct 30 12:35:16.686395 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.686369 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa641d6-2556-4e48-9503-85d33c43df8a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j\" (UID: \"9aa641d6-2556-4e48-9503-85d33c43df8a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" Oct 30 12:35:16.718933 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.718893 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms5tc\" (UniqueName: \"kubernetes.io/projected/9aa641d6-2556-4e48-9503-85d33c43df8a-kube-api-access-ms5tc\") pod \"payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j\" (UID: \"9aa641d6-2556-4e48-9503-85d33c43df8a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" Oct 30 12:35:16.928896 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.928868 4413 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 12:35:16.929076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.928914 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" podUID="75e43f0b-df10-4b4e-b7af-67d515178a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 12:35:16.941971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:16.941941 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" Oct 30 12:35:17.068310 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:17.068273 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" event={"ID":"0e06b3dc-2413-4f9f-ba0d-691519d450f1","Type":"ContainerStarted","Data":"95201af750515388c1419366eacbcbe11349a59bee05d1ad6ae6865bc422bf9a"} Oct 30 12:35:17.068448 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:17.068343 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" event={"ID":"0e06b3dc-2413-4f9f-ba0d-691519d450f1","Type":"ContainerStarted","Data":"dd51b0c4d34a149d3817b88e27f4f3ffe57f0975c8f0760531c38745c5ecc798"} Oct 30 12:35:17.121764 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:17.121704 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" podStartSLOduration=2.121674443 podStartE2EDuration="2.121674443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:17.120677006 +0000 UTC m=+2470.230519887" watchObservedRunningTime="2026-10-30 12:35:17.121674443 +0000 UTC m=+2470.231517327" Oct 30 12:35:17.289294 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:17.275206 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j"] Oct 30 12:35:17.289728 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:35:17.289690 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa641d6_2556_4e48_9503_85d33c43df8a.slice/crio-bb5e08473a8492bea608f4335a93d51fc1978b3ac34f5d2a5ee944ac2d3c92c1 WatchSource:0}: Error finding container bb5e08473a8492bea608f4335a93d51fc1978b3ac34f5d2a5ee944ac2d3c92c1: Status 404 returned error can't find the container with id bb5e08473a8492bea608f4335a93d51fc1978b3ac34f5d2a5ee944ac2d3c92c1 Oct 30 12:35:17.866271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:17.865830 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:17.871777 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:17.871755 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:17.928977 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:17.928940 4413 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 12:35:17.929152 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:17.928995 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" podUID="75e43f0b-df10-4b4e-b7af-67d515178a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 12:35:18.077111 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:18.077008 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" event={"ID":"9aa641d6-2556-4e48-9503-85d33c43df8a","Type":"ContainerStarted","Data":"505bbe69e570b309b94ca53bf22dd8b49763e6d65deae1c45bb13e0533ec59fb"} Oct 30 12:35:18.077111 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:18.077088 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" event={"ID":"9aa641d6-2556-4e48-9503-85d33c43df8a","Type":"ContainerStarted","Data":"bb5e08473a8492bea608f4335a93d51fc1978b3ac34f5d2a5ee944ac2d3c92c1"} Oct 30 12:35:18.077406 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:18.077340 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" Oct 30 12:35:18.078961 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:18.077859 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:18.079193 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:18.079173 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:35:18.120493 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:18.120385 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" podStartSLOduration=2.120365617 podStartE2EDuration="2.120365617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:18.116198509 +0000 UTC m=+2471.226041391" watchObservedRunningTime="2026-10-30 12:35:18.120365617 +0000 UTC m=+2471.230208501" Oct 30 12:35:18.398297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:18.398267 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:18.933324 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:18.933298 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:35:18.934473 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:18.934454 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 12:35:19.808905 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:19.808857 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:19.951434 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:19.951404 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:20.475517 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:20.475486 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:21.466923 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:21.466890 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:21.592139 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:21.592099 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:21.628501 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:21.628471 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4"] Oct 30 12:35:21.631840 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:21.631817 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" Oct 30 12:35:21.662585 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:21.662555 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-42220c-dockercfg-5m4qq\"" Oct 30 12:35:21.680307 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:21.671074 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-42220c\"" Oct 30 12:35:21.684802 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:21.684735 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4"] Oct 30 12:35:21.687828 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:21.687801 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grfvx\" (UniqueName: \"kubernetes.io/projected/70146efe-2585-4812-8761-50cd5c027a1c-kube-api-access-grfvx\") pod \"payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4\" (UID: \"70146efe-2585-4812-8761-50cd5c027a1c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" Oct 30 12:35:21.688088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:21.688068 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70146efe-2585-4812-8761-50cd5c027a1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4\" (UID: \"70146efe-2585-4812-8761-50cd5c027a1c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" Oct 30 12:35:21.845026 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:21.844923 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70146efe-2585-4812-8761-50cd5c027a1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4\" (UID: \"70146efe-2585-4812-8761-50cd5c027a1c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" Oct 30 12:35:21.846817 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:21.846797 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grfvx\" (UniqueName: \"kubernetes.io/projected/70146efe-2585-4812-8761-50cd5c027a1c-kube-api-access-grfvx\") pod \"payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4\" (UID: \"70146efe-2585-4812-8761-50cd5c027a1c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" Oct 30 12:35:21.847335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:21.846315 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70146efe-2585-4812-8761-50cd5c027a1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4\" (UID: \"70146efe-2585-4812-8761-50cd5c027a1c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" Oct 30 12:35:21.870304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:21.870274 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grfvx\" (UniqueName: \"kubernetes.io/projected/70146efe-2585-4812-8761-50cd5c027a1c-kube-api-access-grfvx\") pod \"payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4\" (UID: \"70146efe-2585-4812-8761-50cd5c027a1c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" Oct 30 12:35:21.970738 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:21.970709 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" Oct 30 12:35:22.274407 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:22.274372 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4"] Oct 30 12:35:22.274603 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:35:22.274579 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70146efe_2585_4812_8761_50cd5c027a1c.slice/crio-501f3415a2ff11a166ed0cb61cb8ccd898c4bd15a53f1d09c5d65899111a826e WatchSource:0}: Error finding container 501f3415a2ff11a166ed0cb61cb8ccd898c4bd15a53f1d09c5d65899111a826e: Status 404 returned error can't find the container with id 501f3415a2ff11a166ed0cb61cb8ccd898c4bd15a53f1d09c5d65899111a826e Oct 30 12:35:22.448196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:22.448161 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:23.121037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:23.120995 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" event={"ID":"70146efe-2585-4812-8761-50cd5c027a1c","Type":"ContainerStarted","Data":"7d87ddf9a1b368f6cda798187caf2d874b97e15c8a8de581f31c0360844c55d6"} Oct 30 12:35:23.122154 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:23.121093 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" event={"ID":"70146efe-2585-4812-8761-50cd5c027a1c","Type":"ContainerStarted","Data":"501f3415a2ff11a166ed0cb61cb8ccd898c4bd15a53f1d09c5d65899111a826e"} Oct 30 12:35:23.122154 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:23.122178 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" Oct 30 12:35:23.164526 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:23.164420 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" podStartSLOduration=2.16440537 podStartE2EDuration="2.16440537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:23.162867273 +0000 UTC m=+2476.272710155" watchObservedRunningTime="2026-10-30 12:35:23.16440537 +0000 UTC m=+2476.274248251" Oct 30 12:35:23.916726 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:23.916659 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:24.391097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:24.391057 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:24.502112 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:24.502020 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:24.852908 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:24.849754 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj"] Oct 30 12:35:24.868766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:24.867697 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:24.911284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:24.910155 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-dbe123-openshift-default-dockercfg-xkks6\"" Oct 30 12:35:24.933645 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:24.933610 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:24.935519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:24.935485 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-workload-certs\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:24.935674 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:24.935552 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a8eae2-4b05-4be5-9231-548a3b651406-istiod-ca-cert\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:24.935736 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:24.935581 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a8eae2-4b05-4be5-9231-548a3b651406-istio-podinfo\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:24.935811 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:24.935793 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crxq6\" (UniqueName: \"kubernetes.io/projected/26a8eae2-4b05-4be5-9231-548a3b651406-kube-api-access-crxq6\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:24.935907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:24.935890 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-workload-socket\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:24.936202 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:24.936180 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-istio-envoy\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:24.936305 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:24.936217 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a8eae2-4b05-4be5-9231-548a3b651406-istio-token\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:24.946277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:24.943633 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-credential-socket\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:24.946277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:24.943817 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-istio-data\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:24.948987 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:24.948946 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj"] Oct 30 12:35:25.178341 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.096979 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-workload-certs\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.184287 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.184228 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:25.222302 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.222267 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a8eae2-4b05-4be5-9231-548a3b651406-istiod-ca-cert\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.222985 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.222697 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a8eae2-4b05-4be5-9231-548a3b651406-istio-podinfo\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.223309 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.223269 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crxq6\" (UniqueName: \"kubernetes.io/projected/26a8eae2-4b05-4be5-9231-548a3b651406-kube-api-access-crxq6\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.223674 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.223636 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-workload-socket\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.224182 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.224145 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-istio-envoy\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.224559 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.224363 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a8eae2-4b05-4be5-9231-548a3b651406-istio-token\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.224765 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.224738 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-credential-socket\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.225037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.225008 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-istio-data\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.234302 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.228495 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-workload-socket\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.272294 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.264182 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-istio-data\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.272294 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.264488 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-credential-socket\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.272294 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.264755 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-workload-certs\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.274406 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.252234 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a8eae2-4b05-4be5-9231-548a3b651406-istiod-ca-cert\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.294291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.280711 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-istio-envoy\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.294291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.281365 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a8eae2-4b05-4be5-9231-548a3b651406-istio-podinfo\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.318281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.314348 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a8eae2-4b05-4be5-9231-548a3b651406-istio-token\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.331098 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.331063 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crxq6\" (UniqueName: \"kubernetes.io/projected/26a8eae2-4b05-4be5-9231-548a3b651406-kube-api-access-crxq6\") pod \"e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.517681 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.517578 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:25.642382 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.642341 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:25.886017 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:35:25.885978 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a8eae2_4b05_4be5_9231_548a3b651406.slice/crio-88706a5f52efbc2fa355ab39db065f469a69ac6c5fc6afb0e6699d8930560912 WatchSource:0}: Error finding container 88706a5f52efbc2fa355ab39db065f469a69ac6c5fc6afb0e6699d8930560912: Status 404 returned error can't find the container with id 88706a5f52efbc2fa355ab39db065f469a69ac6c5fc6afb0e6699d8930560912 Oct 30 12:35:25.887784 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.887616 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj"] Oct 30 12:35:25.936278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.914103 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:35:25.936278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.914203 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:35:25.936278 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:25.914275 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:35:26.115361 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:26.108546 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:26.251392 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:26.227767 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:26.426271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:26.406085 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" event={"ID":"26a8eae2-4b05-4be5-9231-548a3b651406","Type":"ContainerStarted","Data":"3769aa7e42ac9b067d38e25b4ec21788301d4847a24d180db71ffd8250bc865a"} Oct 30 12:35:26.426742 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:26.426716 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" event={"ID":"26a8eae2-4b05-4be5-9231-548a3b651406","Type":"ContainerStarted","Data":"88706a5f52efbc2fa355ab39db065f469a69ac6c5fc6afb0e6699d8930560912"} Oct 30 12:35:26.521430 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:26.521338 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:26.534719 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:26.534656 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" podStartSLOduration=2.534633392 podStartE2EDuration="2.534633392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:26.482627166 +0000 UTC m=+2479.592470050" watchObservedRunningTime="2026-10-30 12:35:26.534633392 +0000 UTC m=+2479.644476274" Oct 30 12:35:26.536462 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:26.536313 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:27.535283 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:27.534293 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:27.860301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:27.860191 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:28.449108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:28.448973 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:28.450470 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:28.450448 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:35:28.869153 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:28.869083 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:29.017273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:29.016799 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:29.086921 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:29.086890 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" Oct 30 12:35:29.752452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:29.752426 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:29.885773 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:29.885743 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:30.037359 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:30.034370 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:30.562566 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:30.562527 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:30.675771 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:30.675731 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:30.743200 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:30.743158 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n"] Oct 30 12:35:30.747612 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:30.747591 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n" Oct 30 12:35:30.766335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:30.766307 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n"] Oct 30 12:35:30.773658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:30.773594 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-dbe123\"" Oct 30 12:35:30.773811 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:30.773728 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-dbe123-dockercfg-ht8jb\"" Oct 30 12:35:30.836753 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:30.836646 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dde17ed-ff85-43dd-8cd7-a702a97bbd3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n\" (UID: \"1dde17ed-ff85-43dd-8cd7-a702a97bbd3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n" Oct 30 12:35:30.837389 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:30.837349 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wsgj\" (UniqueName: \"kubernetes.io/projected/1dde17ed-ff85-43dd-8cd7-a702a97bbd3b-kube-api-access-9wsgj\") pod \"payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n\" (UID: \"1dde17ed-ff85-43dd-8cd7-a702a97bbd3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n" Oct 30 12:35:30.940979 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:30.940680 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wsgj\" (UniqueName: \"kubernetes.io/projected/1dde17ed-ff85-43dd-8cd7-a702a97bbd3b-kube-api-access-9wsgj\") pod \"payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n\" (UID: \"1dde17ed-ff85-43dd-8cd7-a702a97bbd3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n" Oct 30 12:35:30.940979 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:30.940734 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dde17ed-ff85-43dd-8cd7-a702a97bbd3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n\" (UID: \"1dde17ed-ff85-43dd-8cd7-a702a97bbd3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n" Oct 30 12:35:30.941624 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:30.941594 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dde17ed-ff85-43dd-8cd7-a702a97bbd3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n\" (UID: \"1dde17ed-ff85-43dd-8cd7-a702a97bbd3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n" Oct 30 12:35:30.964649 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:30.964616 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wsgj\" (UniqueName: \"kubernetes.io/projected/1dde17ed-ff85-43dd-8cd7-a702a97bbd3b-kube-api-access-9wsgj\") pod \"payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n\" (UID: \"1dde17ed-ff85-43dd-8cd7-a702a97bbd3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n" Oct 30 12:35:31.066923 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:31.066893 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n" Oct 30 12:35:31.379069 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:31.378996 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n"] Oct 30 12:35:31.380383 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:35:31.380353 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dde17ed_ff85_43dd_8cd7_a702a97bbd3b.slice/crio-9332a3e5e2e26d17289521f31e1100a079af39a4afad6171682453848ee9acac WatchSource:0}: Error finding container 9332a3e5e2e26d17289521f31e1100a079af39a4afad6171682453848ee9acac: Status 404 returned error can't find the container with id 9332a3e5e2e26d17289521f31e1100a079af39a4afad6171682453848ee9acac Oct 30 12:35:31.472640 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:31.472592 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n" event={"ID":"1dde17ed-ff85-43dd-8cd7-a702a97bbd3b","Type":"ContainerStarted","Data":"9332a3e5e2e26d17289521f31e1100a079af39a4afad6171682453848ee9acac"} Oct 30 12:35:31.673276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:31.673171 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:32.269483 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:32.269440 4413 patch_prober.go:29] interesting pod/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:35:32.269983 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:32.269513 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" podUID="5b66ed77-d0a7-4180-ba18-8598280715af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:35:32.304320 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:32.304281 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:32.417744 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:32.417706 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:32.483347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:32.483303 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n" event={"ID":"1dde17ed-ff85-43dd-8cd7-a702a97bbd3b","Type":"ContainerStarted","Data":"6f62843a107106796831392e6e14b7916d292b3b2c988e1629c0aeee86e72854"} Oct 30 12:35:32.483565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:32.483551 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n" Oct 30 12:35:32.634291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:32.634212 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n" podStartSLOduration=2.634189338 podStartE2EDuration="2.634189338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:32.524845326 +0000 UTC m=+2485.634688220" watchObservedRunningTime="2026-10-30 12:35:32.634189338 +0000 UTC m=+2485.744032220" Oct 30 12:35:33.640716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:33.640683 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:33.758721 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:33.758678 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:33.885699 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:33.885643 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:34.499527 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:34.499495 4413 generic.go:356] "Generic (PLEG): container finished" podID="5b66ed77-d0a7-4180-ba18-8598280715af" containerID="d5eb27e80038c448c69324e58fde9eb52845e69e86953a36445d70c517601905" exitCode=0 Oct 30 12:35:34.500656 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:34.499571 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" event={"ID":"5b66ed77-d0a7-4180-ba18-8598280715af","Type":"ContainerDied","Data":"d5eb27e80038c448c69324e58fde9eb52845e69e86953a36445d70c517601905"} Oct 30 12:35:34.500656 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:34.500041 4413 scope.go:122] "RemoveContainer" containerID="d5eb27e80038c448c69324e58fde9eb52845e69e86953a36445d70c517601905" Oct 30 12:35:34.501751 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:34.501724 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:35:34.501815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:34.501795 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:35:34.501855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:34.501828 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:35:35.316856 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:35.315553 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" Oct 30 12:35:35.469308 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:35.469272 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:35.551103 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:35.550053 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" event={"ID":"5b66ed77-d0a7-4180-ba18-8598280715af","Type":"ContainerStarted","Data":"09521b004b5631deadd8b0939e59fe9fa8dc0e65b25934db4edbeb48869526a4"} Oct 30 12:35:35.551954 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:35.551925 4413 generic.go:356] "Generic (PLEG): container finished" podID="6435f44e-980e-4166-868b-ff94fa2010b9" containerID="0b1529c128f54f2748ccf120adb919179f7481237fa77b6e906805a8aadc1ee9" exitCode=0 Oct 30 12:35:35.552093 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:35.551974 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" event={"ID":"6435f44e-980e-4166-868b-ff94fa2010b9","Type":"ContainerDied","Data":"0b1529c128f54f2748ccf120adb919179f7481237fa77b6e906805a8aadc1ee9"} Oct 30 12:35:35.552370 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:35.552356 4413 scope.go:122] "RemoveContainer" containerID="0b1529c128f54f2748ccf120adb919179f7481237fa77b6e906805a8aadc1ee9" Oct 30 12:35:35.553984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:35.553952 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:35:35.554090 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:35.554026 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:35:35.554090 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:35.554058 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:35:35.570556 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:35.570532 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:35:35.570747 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:35.570727 4413 patch_prober.go:29] interesting pod/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 12:35:35.570799 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:35.570783 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" podUID="5b66ed77-d0a7-4180-ba18-8598280715af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 12:35:35.849463 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:35:35.849177 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:35:36.561216 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:36.561179 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" event={"ID":"6435f44e-980e-4166-868b-ff94fa2010b9","Type":"ContainerStarted","Data":"e870bee964d94c7a593084b16f12abb8b855bd03eb8cb5f101b850720897f9f9"} Oct 30 12:35:36.570228 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:36.570208 4413 patch_prober.go:29] interesting pod/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 12:35:36.570375 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:36.570286 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" podUID="5b66ed77-d0a7-4180-ba18-8598280715af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 12:35:36.649627 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:36.649595 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:35:36.651023 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:36.651001 4413 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 12:35:36.651135 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:36.651057 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" podUID="6435f44e-980e-4166-868b-ff94fa2010b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 12:35:36.842419 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:36.842338 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:35:37.246907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:37.246872 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:37.570540 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:37.570458 4413 patch_prober.go:29] interesting pod/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 12:35:37.570995 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:37.570968 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" podUID="5b66ed77-d0a7-4180-ba18-8598280715af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 12:35:37.650516 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:37.650477 4413 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 12:35:37.650701 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:37.650527 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" podUID="6435f44e-980e-4166-868b-ff94fa2010b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 12:35:38.570357 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:38.570313 4413 patch_prober.go:29] interesting pod/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 12:35:38.570554 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:38.570380 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" podUID="5b66ed77-d0a7-4180-ba18-8598280715af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 12:35:38.650656 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:38.650618 4413 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 12:35:38.650810 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:38.650677 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" podUID="6435f44e-980e-4166-868b-ff94fa2010b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 12:35:38.927677 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:38.924437 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt"] Oct 30 12:35:38.944279 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:38.938625 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" Oct 30 12:35:38.944279 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:38.939195 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt"] Oct 30 12:35:38.960981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:38.960957 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-42220c-serving-cert\"" Oct 30 12:35:38.983453 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:38.983422 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:38.983898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:38.983860 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951adfc0-673a-4716-b07a-e94182a7e608-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt\" (UID: \"951adfc0-673a-4716-b07a-e94182a7e608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" Oct 30 12:35:38.984027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:38.983980 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/951adfc0-673a-4716-b07a-e94182a7e608-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt\" (UID: \"951adfc0-673a-4716-b07a-e94182a7e608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" Oct 30 12:35:38.984436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:38.984414 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951adfc0-673a-4716-b07a-e94182a7e608-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt\" (UID: \"951adfc0-673a-4716-b07a-e94182a7e608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" Oct 30 12:35:38.984660 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:38.984636 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh6gq\" (UniqueName: \"kubernetes.io/projected/951adfc0-673a-4716-b07a-e94182a7e608-kube-api-access-kh6gq\") pod \"maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt\" (UID: \"951adfc0-673a-4716-b07a-e94182a7e608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" Oct 30 12:35:39.086420 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:39.086394 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951adfc0-673a-4716-b07a-e94182a7e608-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt\" (UID: \"951adfc0-673a-4716-b07a-e94182a7e608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" Oct 30 12:35:39.086643 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:39.086628 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh6gq\" (UniqueName: \"kubernetes.io/projected/951adfc0-673a-4716-b07a-e94182a7e608-kube-api-access-kh6gq\") pod \"maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt\" (UID: \"951adfc0-673a-4716-b07a-e94182a7e608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" Oct 30 12:35:39.086859 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:39.086844 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951adfc0-673a-4716-b07a-e94182a7e608-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt\" (UID: \"951adfc0-673a-4716-b07a-e94182a7e608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" Oct 30 12:35:39.086991 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:39.086981 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/951adfc0-673a-4716-b07a-e94182a7e608-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt\" (UID: \"951adfc0-673a-4716-b07a-e94182a7e608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" Oct 30 12:35:39.087410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:39.087388 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951adfc0-673a-4716-b07a-e94182a7e608-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt\" (UID: \"951adfc0-673a-4716-b07a-e94182a7e608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" Oct 30 12:35:39.089138 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:39.089116 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951adfc0-673a-4716-b07a-e94182a7e608-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt\" (UID: \"951adfc0-673a-4716-b07a-e94182a7e608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" Oct 30 12:35:39.089585 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:39.089568 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/951adfc0-673a-4716-b07a-e94182a7e608-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt\" (UID: \"951adfc0-673a-4716-b07a-e94182a7e608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" Oct 30 12:35:39.146824 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:39.146739 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:39.169315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:39.169283 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh6gq\" (UniqueName: \"kubernetes.io/projected/951adfc0-673a-4716-b07a-e94182a7e608-kube-api-access-kh6gq\") pod \"maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt\" (UID: \"951adfc0-673a-4716-b07a-e94182a7e608\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" Oct 30 12:35:39.269756 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:39.269669 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" Oct 30 12:35:39.571100 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:39.571013 4413 patch_prober.go:29] interesting pod/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 12:35:39.571100 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:39.571066 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" podUID="5b66ed77-d0a7-4180-ba18-8598280715af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 12:35:39.642486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:39.642445 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt"] Oct 30 12:35:39.647173 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:35:39.647145 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951adfc0_673a_4716_b07a_e94182a7e608.slice/crio-7e6481676d8371ab84c42f65e9138e146d7bb47779f4c335df3d029605329346 WatchSource:0}: Error finding container 7e6481676d8371ab84c42f65e9138e146d7bb47779f4c335df3d029605329346: Status 404 returned error can't find the container with id 7e6481676d8371ab84c42f65e9138e146d7bb47779f4c335df3d029605329346 Oct 30 12:35:39.650407 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:39.650388 4413 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 12:35:39.650471 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:39.650433 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" podUID="6435f44e-980e-4166-868b-ff94fa2010b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 12:35:40.570825 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:40.570789 4413 patch_prober.go:29] interesting pod/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 12:35:40.571001 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:40.570840 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" podUID="5b66ed77-d0a7-4180-ba18-8598280715af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 12:35:40.627010 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:40.626957 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" event={"ID":"951adfc0-673a-4716-b07a-e94182a7e608","Type":"ContainerStarted","Data":"08352913c8cf32e6ea451ba4708321fe79b1a9c6126fb8a76e946fdd3fcf80f0"} Oct 30 12:35:40.627534 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:40.627083 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" event={"ID":"951adfc0-673a-4716-b07a-e94182a7e608","Type":"ContainerStarted","Data":"7e6481676d8371ab84c42f65e9138e146d7bb47779f4c335df3d029605329346"} Oct 30 12:35:40.627534 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:40.627430 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" Oct 30 12:35:40.650279 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:40.650234 4413 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 12:35:40.650454 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:40.650296 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" podUID="6435f44e-980e-4166-868b-ff94fa2010b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 12:35:40.680899 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:40.680833 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" podStartSLOduration=2.273150003 podStartE2EDuration="2.680812698s" totalImagesPullingTime="407.662695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:40.676207419 +0000 UTC m=+2493.786050301" watchObservedRunningTime="2026-10-30 12:35:40.680812698 +0000 UTC m=+2493.790655580" Oct 30 12:35:40.776169 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:40.776131 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:41.546564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:41.546529 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:41.574372 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:41.574346 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:35:41.574824 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:41.574805 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:35:41.575422 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:41.575409 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 12:35:41.650288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:41.650222 4413 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 12:35:41.650671 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:41.650297 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" podUID="6435f44e-980e-4166-868b-ff94fa2010b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 12:35:42.051337 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.051298 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:42.402812 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.402780 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:42.514282 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.514218 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh"] Oct 30 12:35:42.517904 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.517880 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.545566 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.544968 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-8d2170-openshift-default-dockercfg-4vdd2\"" Oct 30 12:35:42.549134 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.549097 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh"] Oct 30 12:35:42.573133 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.573102 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:42.609546 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.609503 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-workload-certs\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.609546 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.609540 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-podinfo\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.609770 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.609622 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-workload-socket\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.609770 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.609716 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-data\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.609770 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.609738 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istiod-ca-cert\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.609890 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.609788 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-credential-socket\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.609890 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.609874 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7684\" (UniqueName: \"kubernetes.io/projected/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-kube-api-access-m7684\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.609975 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.609928 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-envoy\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.610012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.609982 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-token\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.678663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.678578 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:35:42.703717 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.703674 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 12:35:42.717441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.711986 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-data\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.717441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.712060 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istiod-ca-cert\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.717441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.712174 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-credential-socket\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.717441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.712262 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7684\" (UniqueName: \"kubernetes.io/projected/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-kube-api-access-m7684\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.717441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.712348 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-envoy\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.717441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.712411 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-token\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.717441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.714319 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istiod-ca-cert\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.717441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.714578 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-workload-certs\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.717441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.714624 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-podinfo\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.717441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.714680 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-workload-socket\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.717441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.715003 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-credential-socket\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.717441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.715303 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-data\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.718806 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.718777 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-workload-certs\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.720685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.719828 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-workload-socket\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.744813 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.744773 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-envoy\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.749304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.748714 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-token\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.749304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.749233 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7684\" (UniqueName: \"kubernetes.io/projected/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-kube-api-access-m7684\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.758043 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.758008 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-podinfo\") pod \"e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.832187 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.832146 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:42.942412 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:42.942325 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:43.164572 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:43.164541 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh"] Oct 30 12:35:43.165650 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:35:43.165615 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb76aa8_2ad3_4bb6_bea9_ae31de43322e.slice/crio-8a5125b03ecd40134ecee583df92c25b0e97c95ba93d2ce75a27c9b797af8486 WatchSource:0}: Error finding container 8a5125b03ecd40134ecee583df92c25b0e97c95ba93d2ce75a27c9b797af8486: Status 404 returned error can't find the container with id 8a5125b03ecd40134ecee583df92c25b0e97c95ba93d2ce75a27c9b797af8486 Oct 30 12:35:43.168370 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:43.168341 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:35:43.168440 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:43.168402 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:35:43.168440 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:43.168432 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:35:43.494098 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:43.494009 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n" Oct 30 12:35:43.725911 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:43.725874 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" event={"ID":"adb76aa8-2ad3-4bb6-bea9-ae31de43322e","Type":"ContainerStarted","Data":"926a7ddd3d6509634203cb6dd624ae724b6c25581ef8039181b56296a4d0984b"} Oct 30 12:35:43.726337 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:43.725940 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" event={"ID":"adb76aa8-2ad3-4bb6-bea9-ae31de43322e","Type":"ContainerStarted","Data":"8a5125b03ecd40134ecee583df92c25b0e97c95ba93d2ce75a27c9b797af8486"} Oct 30 12:35:43.775841 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:43.775730 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" podStartSLOduration=1.7757102649999998 podStartE2EDuration="1.775710265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:43.772963382 +0000 UTC m=+2496.882806264" watchObservedRunningTime="2026-10-30 12:35:43.775710265 +0000 UTC m=+2496.885553145" Oct 30 12:35:43.832952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:43.832916 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:44.349134 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:44.349101 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:44.892330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:44.892302 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:45.772133 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:45.772102 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:45.776494 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:45.776468 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:35:46.517607 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:46.517573 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:46.635690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:46.635656 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:46.648449 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:46.648424 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" Oct 30 12:35:48.183555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:48.183516 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:48.298654 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:48.298618 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:48.540319 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:48.540200 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:50.451161 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:50.451095 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:50.565763 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:50.565728 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:50.694730 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:50.694695 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:52.250841 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:52.250812 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:53.714745 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.714712 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b"] Oct 30 12:35:53.716959 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.716930 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:53.719367 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.719351 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" Oct 30 12:35:53.734706 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.734676 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-5d90d8-serving-cert\"" Oct 30 12:35:53.752598 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.752568 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b"] Oct 30 12:35:53.803977 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.803948 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-metrics-certs\") pod \"maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b\" (UID: \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" Oct 30 12:35:53.804172 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.804152 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-maas-api-tls\") pod \"maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b\" (UID: \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" Oct 30 12:35:53.804314 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.804294 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clg97\" (UniqueName: \"kubernetes.io/projected/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-kube-api-access-clg97\") pod \"maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b\" (UID: \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" Oct 30 12:35:53.804358 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.804346 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-openshift-service-ca\") pod \"maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b\" (UID: \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" Oct 30 12:35:53.838677 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.838636 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:53.916157 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.916118 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-maas-api-tls\") pod \"maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b\" (UID: \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" Oct 30 12:35:53.917359 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.917320 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clg97\" (UniqueName: \"kubernetes.io/projected/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-kube-api-access-clg97\") pod \"maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b\" (UID: \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" Oct 30 12:35:53.917532 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.917409 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-openshift-service-ca\") pod \"maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b\" (UID: \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" Oct 30 12:35:53.917532 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.917492 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-metrics-certs\") pod \"maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b\" (UID: \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" Oct 30 12:35:53.920673 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.920640 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-openshift-service-ca\") pod \"maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b\" (UID: \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" Oct 30 12:35:53.924587 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.924556 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-maas-api-tls\") pod \"maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b\" (UID: \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" Oct 30 12:35:53.928720 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.928697 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-metrics-certs\") pod \"maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b\" (UID: \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" Oct 30 12:35:53.942281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.942234 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clg97\" (UniqueName: \"kubernetes.io/projected/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-kube-api-access-clg97\") pod \"maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b\" (UID: \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" Oct 30 12:35:53.951244 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:53.951215 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:54.033836 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:54.033744 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" Oct 30 12:35:54.345992 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:54.345964 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b"] Oct 30 12:35:54.346783 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:35:54.346758 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138fd1f5_aa8f_4bc9_bd63_2fe6cb2e1ae5.slice/crio-bcd4471012718ae4bad0ad3d768414516c85a61600e831fca85dee8353939e50 WatchSource:0}: Error finding container bcd4471012718ae4bad0ad3d768414516c85a61600e831fca85dee8353939e50: Status 404 returned error can't find the container with id bcd4471012718ae4bad0ad3d768414516c85a61600e831fca85dee8353939e50 Oct 30 12:35:54.859492 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:54.859458 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" event={"ID":"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5","Type":"ContainerStarted","Data":"f61c0036a1bb8a034d818a23b24e6f5e54a9ab3ab5160e190d3b4b67ce664dce"} Oct 30 12:35:54.859945 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:54.859511 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" event={"ID":"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5","Type":"ContainerStarted","Data":"bcd4471012718ae4bad0ad3d768414516c85a61600e831fca85dee8353939e50"} Oct 30 12:35:54.859945 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:54.859619 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" Oct 30 12:35:54.908297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:54.908217 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" podStartSLOduration=1.565784517 podStartE2EDuration="1.90820311s" totalImagesPullingTime="342.418593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:35:54.907828286 +0000 UTC m=+2508.017671167" watchObservedRunningTime="2026-10-30 12:35:54.90820311 +0000 UTC m=+2508.018045990" Oct 30 12:35:55.577069 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:55.577033 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:57.320657 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:57.320616 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:58.801549 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:58.801519 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:35:59.098820 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:35:59.098746 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:00.872380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:00.872353 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" Oct 30 12:36:00.974450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:00.974423 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:02.982836 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:02.982796 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:04.679814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:04.679784 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:05.074576 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:05.074498 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:05.814274 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:36:05.814204 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:36:06.770922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:06.770885 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:06.940963 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:06.940925 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:08.652195 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:08.652153 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:10.223276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.215894 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:10.484748 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.484673 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985"] Oct 30 12:36:10.489472 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.489451 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.553710 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.553643 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.554074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.554048 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.554354 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.554333 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.554567 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.554550 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.554857 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.554809 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.555046 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.555027 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cbdc0e-806c-47f0-81dc-f09209554793-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.555240 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.555220 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.555480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.555462 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.555653 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.555634 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlngn\" (UniqueName: \"kubernetes.io/projected/b3cbdc0e-806c-47f0-81dc-f09209554793-kube-api-access-wlngn\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.561635 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.561613 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985"] Oct 30 12:36:10.614275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.598546 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-9d45fb-openshift-default-dockercfg-xd9x9\"" Oct 30 12:36:10.657455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.657424 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.657455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.657459 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cbdc0e-806c-47f0-81dc-f09209554793-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.657669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.657502 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.657669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.657528 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.657669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.657546 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlngn\" (UniqueName: \"kubernetes.io/projected/b3cbdc0e-806c-47f0-81dc-f09209554793-kube-api-access-wlngn\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.657669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.657622 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.657669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.657661 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.657899 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.657724 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.657899 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.657753 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.658321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.658122 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.658321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.658162 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.658509 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.658425 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.658509 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.658456 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.658632 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.658614 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cbdc0e-806c-47f0-81dc-f09209554793-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.660622 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.660603 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.662470 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.662447 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.681438 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.681414 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.681552 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.681536 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlngn\" (UniqueName: \"kubernetes.io/projected/b3cbdc0e-806c-47f0-81dc-f09209554793-kube-api-access-wlngn\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:10.726423 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.726390 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:10.866446 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:10.866344 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:11.243281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:11.243149 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985"] Oct 30 12:36:11.247177 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:36:11.247152 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3cbdc0e_806c_47f0_81dc_f09209554793.slice/crio-84321ad12c6f8f65a6cec41437c351868441aca8fe81d8fa8f1ae4b87abdaf32 WatchSource:0}: Error finding container 84321ad12c6f8f65a6cec41437c351868441aca8fe81d8fa8f1ae4b87abdaf32: Status 404 returned error can't find the container with id 84321ad12c6f8f65a6cec41437c351868441aca8fe81d8fa8f1ae4b87abdaf32 Oct 30 12:36:11.250208 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:11.250177 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:36:11.250314 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:11.250260 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:36:11.250314 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:11.250293 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:36:11.938274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:11.937516 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:12.075874 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:12.075837 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:12.082209 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:12.082181 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" event={"ID":"b3cbdc0e-806c-47f0-81dc-f09209554793","Type":"ContainerStarted","Data":"56b75b93d39bc4fb81f296b4e3be09fbdc0db125411a64f285775c5a7ba536fd"} Oct 30 12:36:12.082373 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:12.082233 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" event={"ID":"b3cbdc0e-806c-47f0-81dc-f09209554793","Type":"ContainerStarted","Data":"84321ad12c6f8f65a6cec41437c351868441aca8fe81d8fa8f1ae4b87abdaf32"} Oct 30 12:36:12.167521 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:12.166312 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" podStartSLOduration=2.166287915 podStartE2EDuration="2.166287915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:12.152963131 +0000 UTC m=+2525.262806016" watchObservedRunningTime="2026-10-30 12:36:12.166287915 +0000 UTC m=+2525.276130796" Oct 30 12:36:12.302908 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:12.302817 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:12.867681 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:12.867595 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:12.869010 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:12.868989 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 12:36:12.869148 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:12.869032 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" podUID="b3cbdc0e-806c-47f0-81dc-f09209554793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 12:36:13.867425 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:13.867386 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 12:36:13.867859 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:13.867451 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" podUID="b3cbdc0e-806c-47f0-81dc-f09209554793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 12:36:13.969987 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:13.969914 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:14.500275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:14.491375 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:36:14.500275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:14.491444 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" podUID="26a8eae2-4b05-4be5-9231-548a3b651406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:36:14.867553 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:14.867464 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 12:36:14.867553 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:14.867537 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" podUID="b3cbdc0e-806c-47f0-81dc-f09209554793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 12:36:15.867651 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:15.867608 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 12:36:15.868095 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:15.867671 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" podUID="b3cbdc0e-806c-47f0-81dc-f09209554793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 12:36:15.965371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:15.965329 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:16.867201 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:16.867158 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 12:36:16.867397 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:16.867226 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" podUID="b3cbdc0e-806c-47f0-81dc-f09209554793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 12:36:17.867587 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:17.867551 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 12:36:17.869452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:17.867630 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" podUID="b3cbdc0e-806c-47f0-81dc-f09209554793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 12:36:17.874423 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:17.874393 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:18.079455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:18.079371 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 12:36:18.079455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:18.079439 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" podUID="0e06b3dc-2413-4f9f-ba0d-691519d450f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 12:36:18.161237 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:18.161196 4413 generic.go:356] "Generic (PLEG): container finished" podID="0e06b3dc-2413-4f9f-ba0d-691519d450f1" containerID="95201af750515388c1419366eacbcbe11349a59bee05d1ad6ae6865bc422bf9a" exitCode=0 Oct 30 12:36:18.161467 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:18.161295 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" event={"ID":"0e06b3dc-2413-4f9f-ba0d-691519d450f1","Type":"ContainerDied","Data":"95201af750515388c1419366eacbcbe11349a59bee05d1ad6ae6865bc422bf9a"} Oct 30 12:36:18.161906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:18.161883 4413 scope.go:122] "RemoveContainer" containerID="95201af750515388c1419366eacbcbe11349a59bee05d1ad6ae6865bc422bf9a" Oct 30 12:36:18.163968 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:18.163927 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:36:18.164078 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:18.164027 4413 generic.go:356] "Generic (PLEG): container finished" podID="26a8eae2-4b05-4be5-9231-548a3b651406" containerID="3769aa7e42ac9b067d38e25b4ec21788301d4847a24d180db71ffd8250bc865a" exitCode=0 Oct 30 12:36:18.164136 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:18.164102 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" event={"ID":"26a8eae2-4b05-4be5-9231-548a3b651406","Type":"ContainerDied","Data":"3769aa7e42ac9b067d38e25b4ec21788301d4847a24d180db71ffd8250bc865a"} Oct 30 12:36:18.164192 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:18.164031 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:36:18.164308 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:18.164281 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:36:18.164481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:18.164468 4413 scope.go:122] "RemoveContainer" containerID="3769aa7e42ac9b067d38e25b4ec21788301d4847a24d180db71ffd8250bc865a" Oct 30 12:36:18.166325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:18.166294 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:36:18.166440 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:18.166373 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:36:18.166440 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:18.166420 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:36:18.872934 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:18.872900 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:18.873447 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:18.873375 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:18.874095 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:18.874079 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:36:19.173285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:19.173096 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" event={"ID":"0e06b3dc-2413-4f9f-ba0d-691519d450f1","Type":"ContainerStarted","Data":"99d1b70dffabf794317ce94654a3e6839a16e16fbe0571dc1ae99d153bfa977d"} Oct 30 12:36:19.175803 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:19.175274 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" event={"ID":"26a8eae2-4b05-4be5-9231-548a3b651406","Type":"ContainerStarted","Data":"3e5579df864ef5fddf99fe1cb3d41028e4614c558d73135fa28c135321fb88bf"} Oct 30 12:36:19.519916 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:19.519826 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:36:19.520311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:19.520282 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 12:36:19.520540 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:19.520514 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" podUID="26a8eae2-4b05-4be5-9231-548a3b651406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 12:36:19.634117 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:19.634071 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:19.865441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:19.865356 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:36:19.865441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:19.865388 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 12:36:19.865641 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:19.865444 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" podUID="0e06b3dc-2413-4f9f-ba0d-691519d450f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 12:36:20.524279 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:20.522346 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 12:36:20.524279 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:20.522412 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" podUID="26a8eae2-4b05-4be5-9231-548a3b651406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 12:36:20.865425 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:20.865329 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 12:36:20.865425 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:20.865397 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" podUID="0e06b3dc-2413-4f9f-ba0d-691519d450f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 12:36:21.438275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:21.433047 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:21.520720 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:21.520682 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 12:36:21.521014 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:21.520981 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" podUID="26a8eae2-4b05-4be5-9231-548a3b651406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 12:36:21.866315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:21.866210 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 12:36:21.866315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:21.866302 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" podUID="0e06b3dc-2413-4f9f-ba0d-691519d450f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 12:36:22.524287 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:22.522416 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 12:36:22.524287 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:22.522476 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" podUID="26a8eae2-4b05-4be5-9231-548a3b651406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 12:36:22.868516 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:22.868414 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 12:36:22.877413 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:22.877358 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" podUID="0e06b3dc-2413-4f9f-ba0d-691519d450f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 12:36:23.220276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:23.219979 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:23.524331 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:23.523946 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 12:36:23.524331 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:23.524011 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" podUID="26a8eae2-4b05-4be5-9231-548a3b651406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 12:36:23.865901 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:23.865812 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 12:36:23.865901 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:23.865865 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" podUID="0e06b3dc-2413-4f9f-ba0d-691519d450f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 12:36:24.709829 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:24.709794 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:36:24.710980 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:24.710957 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:36:24.711735 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:24.711715 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 12:36:24.871833 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:24.871805 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:36:24.872124 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:24.872095 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:36:24.873014 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:24.872994 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 12:36:24.926624 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:24.926589 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:25.124224 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:25.124186 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:26.002136 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:26.002106 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:26.666047 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:26.666015 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:28.314590 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:28.314556 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:29.867195 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:29.867129 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:30.417336 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:30.417300 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:31.550405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:31.550359 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:33.416517 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:33.416479 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:35.116670 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:35.116637 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:35.842572 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:36:35.842519 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:36:36.545654 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:36.545617 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:36.669782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:36.669750 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:38.462217 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:38.462184 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:40.132848 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:40.132784 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:41.845045 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:41.844978 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:42.513018 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:42.512930 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:43.387636 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:43.387606 4413 generic.go:356] "Generic (PLEG): container finished" podID="adb76aa8-2ad3-4bb6-bea9-ae31de43322e" containerID="926a7ddd3d6509634203cb6dd624ae724b6c25581ef8039181b56296a4d0984b" exitCode=0 Oct 30 12:36:43.388086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:43.387642 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" event={"ID":"adb76aa8-2ad3-4bb6-bea9-ae31de43322e","Type":"ContainerDied","Data":"926a7ddd3d6509634203cb6dd624ae724b6c25581ef8039181b56296a4d0984b"} Oct 30 12:36:43.388138 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:43.388120 4413 scope.go:122] "RemoveContainer" containerID="926a7ddd3d6509634203cb6dd624ae724b6c25581ef8039181b56296a4d0984b" Oct 30 12:36:43.389888 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:43.389851 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:36:43.390012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:43.389928 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:36:43.390012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:43.389964 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:36:43.566862 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:43.566826 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:44.403142 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:44.401920 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" event={"ID":"adb76aa8-2ad3-4bb6-bea9-ae31de43322e","Type":"ContainerStarted","Data":"21c846ad397bb2cc4fc7816ee0ece9a035a4200cedd7662c2acbb9dcbd21929f"} Oct 30 12:36:44.833203 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:44.833108 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:36:44.834542 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:44.834519 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 12:36:44.834606 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:44.834575 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" podUID="adb76aa8-2ad3-4bb6-bea9-ae31de43322e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 12:36:45.247386 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:45.247349 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:45.772162 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:45.772124 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:36:45.833148 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:45.833119 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 12:36:45.833345 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:45.833168 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" podUID="adb76aa8-2ad3-4bb6-bea9-ae31de43322e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 12:36:46.832858 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:46.832821 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 12:36:46.833374 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:46.832879 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" podUID="adb76aa8-2ad3-4bb6-bea9-ae31de43322e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 12:36:46.874578 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:46.874549 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:47.832885 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:47.832856 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 12:36:47.833311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:47.832907 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" podUID="adb76aa8-2ad3-4bb6-bea9-ae31de43322e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 12:36:48.840286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:48.834727 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 12:36:48.840286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:48.834792 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" podUID="adb76aa8-2ad3-4bb6-bea9-ae31de43322e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 12:36:48.896078 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:48.896045 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:49.832846 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:49.832797 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 12:36:49.833072 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:49.832862 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" podUID="adb76aa8-2ad3-4bb6-bea9-ae31de43322e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 12:36:50.564819 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:50.564775 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:50.680979 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:50.680948 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:50.837982 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:50.837917 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:36:50.838971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:50.838954 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 12:36:52.134625 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:52.134592 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:52.284738 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:52.284703 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:53.989618 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:53.989587 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:55.389155 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.389122 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8"] Oct 30 12:36:55.393108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.393087 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.409650 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.409630 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9d45fb\"/\"kube-root-ca.crt\"" Oct 30 12:36:55.421174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.421148 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9d45fb\"/\"openshift-service-ca.crt\"" Oct 30 12:36:55.421364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.421198 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9d45fb\"/\"default-dockercfg-ww9j7\"" Oct 30 12:36:55.421447 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.421368 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9d45fb\"/\"auth-test-model-9d45fb-kserve-self-signed-certs\"" Oct 30 12:36:55.424875 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.424851 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8"] Oct 30 12:36:55.446436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.446412 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:55.520620 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.520591 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/676182ad-449e-49da-822a-a4af9861f76b-tls-certs\") pod \"auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.520787 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.520655 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-home\") pod \"auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.520787 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.520742 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-model-cache\") pod \"auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.520910 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.520892 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-tmp-dir\") pod \"auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.520996 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.520979 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvddz\" (UniqueName: \"kubernetes.io/projected/676182ad-449e-49da-822a-a4af9861f76b-kube-api-access-kvddz\") pod \"auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.521135 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.521120 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-dshm\") pod \"auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.608392 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.608359 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6"] Oct 30 12:36:55.612194 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.612175 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.622641 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.622612 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-dshm\") pod \"auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.622754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.622684 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/676182ad-449e-49da-822a-a4af9861f76b-tls-certs\") pod \"auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.622813 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.622771 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-home\") pod \"auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.622865 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.622808 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-model-cache\") pod \"auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.622968 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.622923 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-tmp-dir\") pod \"auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.623110 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.622995 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvddz\" (UniqueName: \"kubernetes.io/projected/676182ad-449e-49da-822a-a4af9861f76b-kube-api-access-kvddz\") pod \"auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.623923 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.623858 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-home\") pod \"auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.623923 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.623872 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-model-cache\") pod \"auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.624077 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.623987 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-tmp-dir\") pod \"auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.626978 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.626956 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-dshm\") pod \"auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.638708 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.628228 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/676182ad-449e-49da-822a-a4af9861f76b-tls-certs\") pod \"auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.655305 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.650754 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvddz\" (UniqueName: \"kubernetes.io/projected/676182ad-449e-49da-822a-a4af9861f76b-kube-api-access-kvddz\") pod \"auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.661710 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.661672 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6"] Oct 30 12:36:55.663375 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.663351 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:55.724330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.711627 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:55.750114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.746548 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpxp4\" (UniqueName: \"kubernetes.io/projected/fd580504-41ed-4b72-b393-62d3cc8a49b2-kube-api-access-rpxp4\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.750114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.746676 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.750114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.746824 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.750114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.746872 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.750114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.747221 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.750114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.747274 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.750114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.747372 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.750114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.747431 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.750114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.747498 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd580504-41ed-4b72-b393-62d3cc8a49b2-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.849179 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.849137 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.849341 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.849193 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.849341 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.849280 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.849341 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.849324 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.849529 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.849362 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd580504-41ed-4b72-b393-62d3cc8a49b2-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.849529 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.849427 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpxp4\" (UniqueName: \"kubernetes.io/projected/fd580504-41ed-4b72-b393-62d3cc8a49b2-kube-api-access-rpxp4\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.849529 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.849493 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.849690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.849600 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.849690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.849631 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.850328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.850299 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-workload-certs\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.850989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.850589 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-workload-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.850989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.850855 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-data\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.851148 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.851124 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-credential-socket\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.853348 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.853319 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd580504-41ed-4b72-b393-62d3cc8a49b2-istiod-ca-cert\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.855667 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.855635 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-envoy\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.856413 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.856388 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-podinfo\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.874283 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.874263 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-token\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.874680 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.874665 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpxp4\" (UniqueName: \"kubernetes.io/projected/fd580504-41ed-4b72-b393-62d3cc8a49b2-kube-api-access-rpxp4\") pod \"e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:55.930834 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:55.930781 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:56.089908 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:56.089882 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8"] Oct 30 12:36:56.090146 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:36:56.090125 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod676182ad_449e_49da_822a_a4af9861f76b.slice/crio-b45a56a492a71d32e58dd106917b962bf21eb417117c35cc816cb6e2f1794a38 WatchSource:0}: Error finding container b45a56a492a71d32e58dd106917b962bf21eb417117c35cc816cb6e2f1794a38: Status 404 returned error can't find the container with id b45a56a492a71d32e58dd106917b962bf21eb417117c35cc816cb6e2f1794a38 Oct 30 12:36:56.241675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:56.241643 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6"] Oct 30 12:36:56.242605 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:36:56.242580 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd580504_41ed_4b72_b393_62d3cc8a49b2.slice/crio-94b8c2def249b4e1da9c1d675c416758509571898a7e56589985b48eeabeb5f0 WatchSource:0}: Error finding container 94b8c2def249b4e1da9c1d675c416758509571898a7e56589985b48eeabeb5f0: Status 404 returned error can't find the container with id 94b8c2def249b4e1da9c1d675c416758509571898a7e56589985b48eeabeb5f0 Oct 30 12:36:56.245301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:56.245234 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:36:56.245366 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:56.245347 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:36:56.245414 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:56.245397 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:36:56.498975 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:56.498883 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" event={"ID":"676182ad-449e-49da-822a-a4af9861f76b","Type":"ContainerStarted","Data":"b34f8762e6b88323b2367c64b6b53e8563d8e8455d01436205cf8efcb496dd38"} Oct 30 12:36:56.498975 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:56.498930 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" event={"ID":"676182ad-449e-49da-822a-a4af9861f76b","Type":"ContainerStarted","Data":"b45a56a492a71d32e58dd106917b962bf21eb417117c35cc816cb6e2f1794a38"} Oct 30 12:36:56.500806 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:56.500778 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" event={"ID":"fd580504-41ed-4b72-b393-62d3cc8a49b2","Type":"ContainerStarted","Data":"e62bc2310b7c70570b60fc5548e28aae393f208d016b46028a3e8ad3924a9afb"} Oct 30 12:36:56.500922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:56.500831 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" event={"ID":"fd580504-41ed-4b72-b393-62d3cc8a49b2","Type":"ContainerStarted","Data":"94b8c2def249b4e1da9c1d675c416758509571898a7e56589985b48eeabeb5f0"} Oct 30 12:36:56.545831 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:56.545774 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" podStartSLOduration=1.5457557259999999 podStartE2EDuration="1.545755726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:56.543040649 +0000 UTC m=+2569.652883532" watchObservedRunningTime="2026-10-30 12:36:56.545755726 +0000 UTC m=+2569.655598608" Oct 30 12:36:56.589571 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:56.589373 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" podStartSLOduration=1.589353861 podStartE2EDuration="1.589353861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:36:56.585839175 +0000 UTC m=+2569.695682055" watchObservedRunningTime="2026-10-30 12:36:56.589353861 +0000 UTC m=+2569.699196742" Oct 30 12:36:56.726228 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:56.725512 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:36:56.931475 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:56.931437 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:36:57.514290 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:57.513954 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:57.933441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:57.933408 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 12:36:57.933643 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:57.933468 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" podUID="fd580504-41ed-4b72-b393-62d3cc8a49b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 12:36:58.921663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:58.921631 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:58.932027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:58.931992 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 12:36:58.932125 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:58.932045 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" podUID="fd580504-41ed-4b72-b393-62d3cc8a49b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 12:36:59.257183 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:59.257097 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:36:59.931704 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:59.931674 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 12:36:59.932105 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:36:59.931722 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" podUID="fd580504-41ed-4b72-b393-62d3cc8a49b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 12:37:00.931506 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:00.931468 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 12:37:00.931694 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:00.931535 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" podUID="fd580504-41ed-4b72-b393-62d3cc8a49b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 12:37:00.950691 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:00.950666 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:01.931474 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:01.931440 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 12:37:01.931803 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:01.931762 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" podUID="fd580504-41ed-4b72-b393-62d3cc8a49b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 12:37:02.666348 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:02.666314 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:02.937074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:02.937001 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:37:02.937444 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:02.937423 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:37:02.938066 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:02.938048 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 12:37:04.354789 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:04.354751 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:05.691164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:05.691129 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:05.726241 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:05.726208 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:37:05.739339 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:05.739312 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:37:05.815366 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:37:05.815331 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:37:06.100675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:06.100587 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:06.609378 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:06.609346 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:37:07.982927 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:07.982872 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:09.796330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:09.796296 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:11.360126 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:11.360084 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:13.041890 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:13.041861 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:14.453394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:14.453358 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:14.748369 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:14.748283 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:16.528940 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:16.528906 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:18.069038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:18.068999 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:18.180038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:18.179992 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:19.954989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:19.954953 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:21.652614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:21.652580 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:23.205463 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:23.205379 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:23.911554 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:23.911521 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:24.023506 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:24.023467 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:24.823154 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:24.823107 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:25.323809 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:25.323777 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:25.440000 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:25.439967 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:26.502125 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:26.502083 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:28.164096 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:28.164061 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:29.757300 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:29.757229 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:31.447536 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:31.447505 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:33.273586 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:33.273517 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:34.912141 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:34.912103 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:35.817165 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:37:35.817127 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:37:36.164585 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.164555 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf"] Oct 30 12:37:36.168144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.168122 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.194040 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.194004 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf"] Oct 30 12:37:36.199653 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.199626 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-dbe123\"/\"auth-test-model-dbe123-kserve-self-signed-certs\"" Oct 30 12:37:36.200500 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.200481 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-dbe123\"/\"kube-root-ca.crt\"" Oct 30 12:37:36.200608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.200593 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-dbe123\"/\"openshift-service-ca.crt\"" Oct 30 12:37:36.200716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.200702 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-dbe123\"/\"default-dockercfg-qljs7\"" Oct 30 12:37:36.232069 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.232039 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:36.305978 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.305943 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-model-cache\") pod \"auth-test-model-dbe123-kserve-865d75f9c-757gf\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.305978 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.305980 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-tmp-dir\") pod \"auth-test-model-dbe123-kserve-865d75f9c-757gf\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.306176 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.306116 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfswt\" (UniqueName: \"kubernetes.io/projected/72a8d280-04de-4a71-8036-dd8775bd9430-kube-api-access-zfswt\") pod \"auth-test-model-dbe123-kserve-865d75f9c-757gf\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.306176 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.306147 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72a8d280-04de-4a71-8036-dd8775bd9430-tls-certs\") pod \"auth-test-model-dbe123-kserve-865d75f9c-757gf\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.306311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.306298 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-dshm\") pod \"auth-test-model-dbe123-kserve-865d75f9c-757gf\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.306352 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.306331 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-home\") pod \"auth-test-model-dbe123-kserve-865d75f9c-757gf\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.407302 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.407271 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-dshm\") pod \"auth-test-model-dbe123-kserve-865d75f9c-757gf\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.407511 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.407331 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-home\") pod \"auth-test-model-dbe123-kserve-865d75f9c-757gf\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.407511 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.407479 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-model-cache\") pod \"auth-test-model-dbe123-kserve-865d75f9c-757gf\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.407511 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.407508 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-tmp-dir\") pod \"auth-test-model-dbe123-kserve-865d75f9c-757gf\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.407702 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.407636 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfswt\" (UniqueName: \"kubernetes.io/projected/72a8d280-04de-4a71-8036-dd8775bd9430-kube-api-access-zfswt\") pod \"auth-test-model-dbe123-kserve-865d75f9c-757gf\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.407756 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.407702 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72a8d280-04de-4a71-8036-dd8775bd9430-tls-certs\") pod \"auth-test-model-dbe123-kserve-865d75f9c-757gf\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.408420 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.408368 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-tmp-dir\") pod \"auth-test-model-dbe123-kserve-865d75f9c-757gf\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.408732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.408449 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-model-cache\") pod \"auth-test-model-dbe123-kserve-865d75f9c-757gf\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.409033 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.409009 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-home\") pod \"auth-test-model-dbe123-kserve-865d75f9c-757gf\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.411469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.411443 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72a8d280-04de-4a71-8036-dd8775bd9430-tls-certs\") pod \"auth-test-model-dbe123-kserve-865d75f9c-757gf\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.412932 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.412882 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-dshm\") pod \"auth-test-model-dbe123-kserve-865d75f9c-757gf\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.455949 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.454410 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfswt\" (UniqueName: \"kubernetes.io/projected/72a8d280-04de-4a71-8036-dd8775bd9430-kube-api-access-zfswt\") pod \"auth-test-model-dbe123-kserve-865d75f9c-757gf\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.473339 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.472359 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:36.483185 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.482939 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:36.794069 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.794042 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf"] Oct 30 12:37:36.795239 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:37:36.795212 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72a8d280_04de_4a71_8036_dd8775bd9430.slice/crio-ed5a94a7e3aae8f5187c0a0f9ed4124aa99ce435d3481aaa2d4bda54f59bf05c WatchSource:0}: Error finding container ed5a94a7e3aae8f5187c0a0f9ed4124aa99ce435d3481aaa2d4bda54f59bf05c: Status 404 returned error can't find the container with id ed5a94a7e3aae8f5187c0a0f9ed4124aa99ce435d3481aaa2d4bda54f59bf05c Oct 30 12:37:36.858001 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:36.857976 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" event={"ID":"72a8d280-04de-4a71-8036-dd8775bd9430","Type":"ContainerStarted","Data":"ed5a94a7e3aae8f5187c0a0f9ed4124aa99ce435d3481aaa2d4bda54f59bf05c"} Oct 30 12:37:37.867237 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:37.867202 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" event={"ID":"72a8d280-04de-4a71-8036-dd8775bd9430","Type":"ContainerStarted","Data":"e894acdf16a4f309a7443d61b101719f27356acf5c907a8d518e48fff1d2b917"} Oct 30 12:37:37.911567 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:37.911511 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" podStartSLOduration=1.911493814 podStartE2EDuration="1.911493814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:37:37.908177828 +0000 UTC m=+2611.018020709" watchObservedRunningTime="2026-10-30 12:37:37.911493814 +0000 UTC m=+2611.021336694" Oct 30 12:37:37.935132 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:37.935100 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b"] Oct 30 12:37:37.935349 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:37.935328 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" podUID="138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5" containerName="maas-api" containerID="cri-o://f61c0036a1bb8a034d818a23b24e6f5e54a9ab3ab5160e190d3b4b67ce664dce" gracePeriod=30 Oct 30 12:37:38.006852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.006815 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:38.290324 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.290303 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" Oct 30 12:37:38.355281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.348124 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-metrics-certs\") pod \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\" (UID: \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\") " Oct 30 12:37:38.355281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.348180 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-kube-api-access-clg97\" (UniqueName: \"kubernetes.io/projected/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-kube-api-access-clg97\") pod \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\" (UID: \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\") " Oct 30 12:37:38.355281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.348323 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-maas-api-tls\") pod \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\" (UID: \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\") " Oct 30 12:37:38.355281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.348371 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-openshift-service-ca\") pod \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\" (UID: \"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5\") " Oct 30 12:37:38.355281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.353469 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-openshift-service-ca" pod "138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5" (UID: "138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:37:38.395274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.379985 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-kube-api-access-clg97" pod "138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5" (UID: "138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5"). InnerVolumeSpecName "kube-api-access-clg97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:37:38.402957 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.402905 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-maas-api-tls" pod "138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5" (UID: "138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:37:38.403699 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.403659 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-metrics-certs" pod "138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5" (UID: "138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:37:38.453777 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.453739 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:37:38.453777 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.453776 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clg97\" (UniqueName: \"kubernetes.io/projected/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-kube-api-access-clg97\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:37:38.453995 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.453790 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:37:38.453995 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.453803 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:37:38.483816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.483772 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:38.580007 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.579906 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:38.698102 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.698063 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:38.814191 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.814159 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:38.875808 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.875774 4413 generic.go:356] "Generic (PLEG): container finished" podID="138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5" containerID="f61c0036a1bb8a034d818a23b24e6f5e54a9ab3ab5160e190d3b4b67ce664dce" exitCode=0 Oct 30 12:37:38.876365 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.875824 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" Oct 30 12:37:38.876365 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.875864 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" event={"ID":"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5","Type":"ContainerDied","Data":"f61c0036a1bb8a034d818a23b24e6f5e54a9ab3ab5160e190d3b4b67ce664dce"} Oct 30 12:37:38.876365 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.875930 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b" event={"ID":"138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5","Type":"ContainerDied","Data":"bcd4471012718ae4bad0ad3d768414516c85a61600e831fca85dee8353939e50"} Oct 30 12:37:38.876365 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.875967 4413 scope.go:122] "RemoveContainer" containerID="f61c0036a1bb8a034d818a23b24e6f5e54a9ab3ab5160e190d3b4b67ce664dce" Oct 30 12:37:38.887185 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.887169 4413 scope.go:122] "RemoveContainer" containerID="f61c0036a1bb8a034d818a23b24e6f5e54a9ab3ab5160e190d3b4b67ce664dce" Oct 30 12:37:38.887518 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:37:38.887498 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61c0036a1bb8a034d818a23b24e6f5e54a9ab3ab5160e190d3b4b67ce664dce\": container with ID starting with f61c0036a1bb8a034d818a23b24e6f5e54a9ab3ab5160e190d3b4b67ce664dce not found: ID does not exist" containerID="f61c0036a1bb8a034d818a23b24e6f5e54a9ab3ab5160e190d3b4b67ce664dce" Oct 30 12:37:38.887601 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.887527 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61c0036a1bb8a034d818a23b24e6f5e54a9ab3ab5160e190d3b4b67ce664dce"} err="failed to get container status \"f61c0036a1bb8a034d818a23b24e6f5e54a9ab3ab5160e190d3b4b67ce664dce\": rpc error: code = NotFound desc = could not find container \"f61c0036a1bb8a034d818a23b24e6f5e54a9ab3ab5160e190d3b4b67ce664dce\": container with ID starting with f61c0036a1bb8a034d818a23b24e6f5e54a9ab3ab5160e190d3b4b67ce664dce not found: ID does not exist" Oct 30 12:37:38.927841 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.927810 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b"] Oct 30 12:37:38.939237 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:38.939207 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5d90d8-74f7fc8ff4-hlt5b"] Oct 30 12:37:39.021117 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:39.021086 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5" path="/var/lib/kubelet/pods/138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5/volumes" Oct 30 12:37:39.260032 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:39.259948 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:40.449868 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:40.449839 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:42.056622 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:42.056589 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:43.602954 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:43.602921 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:45.212987 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:45.212948 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:45.615146 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:45.615051 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:45.741604 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:45.741567 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:46.063604 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:46.063577 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:46.483213 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:46.483181 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:46.495840 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:46.495816 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:46.690680 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:46.690644 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:46.943823 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:46.943796 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:37:47.070677 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:47.070646 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:47.233715 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:47.233631 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:47.578550 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:47.578467 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:48.797942 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:48.797879 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:48.916598 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:48.916569 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:49.114384 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:49.114234 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:50.472709 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:50.472673 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:51.548192 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:51.548163 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:52.278629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:52.278593 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:54.052875 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:54.052845 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:55.757214 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:55.757179 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:57.435461 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:57.435427 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:59.056182 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:59.056146 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:37:59.175093 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:37:59.175046 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:00.698513 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:00.698479 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:02.265402 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:02.265362 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:03.952800 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:03.952762 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:05.530144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:05.530112 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:05.645902 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:05.645866 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:05.830021 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:38:05.829940 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:38:07.173206 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:07.173170 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:08.802819 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:08.802782 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:10.366768 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:10.366726 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:12.054849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:12.054819 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:13.843675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:13.843639 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:15.610945 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:15.610900 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:15.746682 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:15.746650 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:17.098925 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:17.098888 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:18.121534 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:18.121504 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:18.277109 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:18.277054 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:18.625221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:18.625192 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:18.781762 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:18.781729 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:18.983933 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:18.983813 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:20.457886 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:20.457832 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:22.331769 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:22.331736 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:23.914220 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:23.914156 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:24.236958 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:24.236866 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:25.947597 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:25.947565 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:27.545092 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:27.545060 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:29.164710 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:29.164679 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:30.796045 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:30.796016 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:32.438222 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:32.438185 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:34.173009 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:34.172977 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:35.815550 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:38:35.815516 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:38:35.881390 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:35.881362 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:37.575072 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:37.575032 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:39.124704 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:39.124652 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:40.842184 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:40.842153 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:42.563448 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:42.563413 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:44.347942 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:44.347907 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:46.043565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:46.043529 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:47.603560 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:47.603518 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:49.152118 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:49.152082 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:50.761553 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:50.761520 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:52.371448 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:52.371417 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:53.944458 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:53.944421 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:55.498776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:55.498745 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:57.139401 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:57.139363 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:58.680794 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:58.680759 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:38:59.164685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:38:59.164648 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:00.454462 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:00.454415 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:00.576318 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:00.576288 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:01.152727 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:01.152695 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:02.118005 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:02.117969 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:03.741754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:03.741719 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:05.275152 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:05.275124 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:05.814890 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:39:05.814852 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:39:06.429640 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:06.429605 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:06.550312 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:06.549068 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:06.967284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:06.967236 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:08.096098 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:08.096058 4413 scope.go:122] "RemoveContainer" containerID="04c6ed8bfc4803b4e28ed298fddabac7632ff25a87c9795d0d056428877a61aa" Oct 30 12:39:08.602883 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:08.602852 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:10.367074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:10.367044 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:10.771283 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:10.771213 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:10.898764 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:10.898727 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:12.247348 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:12.247312 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:12.444287 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:12.444240 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:12.556154 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:12.556071 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:12.751378 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:12.751324 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7"] Oct 30 12:39:12.753301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:12.753231 4413 cpu_manager.go:414] "containerMap: removing container" podUID="138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5" containerName="maas-api" Oct 30 12:39:12.753527 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:12.753505 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5" containerName="maas-api" Oct 30 12:39:12.753922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:12.753901 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="138fd1f5-aa8f-4bc9-bd63-2fe6cb2e1ae5" containerName="maas-api" Oct 30 12:39:12.789613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:12.789580 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7"] Oct 30 12:39:12.789955 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:12.789939 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7" Oct 30 12:39:12.837510 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:12.834802 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-73bafc\"" Oct 30 12:39:12.837510 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:12.834873 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-73bafc-dockercfg-mqbv4\"" Oct 30 12:39:12.903778 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:12.903745 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b86g9\" (UniqueName: \"kubernetes.io/projected/0c2f60ee-cdf2-4a48-a6ff-092f61a349e2-kube-api-access-b86g9\") pod \"payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7\" (UID: \"0c2f60ee-cdf2-4a48-a6ff-092f61a349e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7" Oct 30 12:39:12.903955 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:12.903818 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2f60ee-cdf2-4a48-a6ff-092f61a349e2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7\" (UID: \"0c2f60ee-cdf2-4a48-a6ff-092f61a349e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7" Oct 30 12:39:13.005165 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:13.005133 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b86g9\" (UniqueName: \"kubernetes.io/projected/0c2f60ee-cdf2-4a48-a6ff-092f61a349e2-kube-api-access-b86g9\") pod \"payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7\" (UID: \"0c2f60ee-cdf2-4a48-a6ff-092f61a349e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7" Oct 30 12:39:13.005393 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:13.005221 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2f60ee-cdf2-4a48-a6ff-092f61a349e2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7\" (UID: \"0c2f60ee-cdf2-4a48-a6ff-092f61a349e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7" Oct 30 12:39:13.005935 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:13.005915 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2f60ee-cdf2-4a48-a6ff-092f61a349e2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7\" (UID: \"0c2f60ee-cdf2-4a48-a6ff-092f61a349e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7" Oct 30 12:39:13.028072 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:13.028046 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b86g9\" (UniqueName: \"kubernetes.io/projected/0c2f60ee-cdf2-4a48-a6ff-092f61a349e2-kube-api-access-b86g9\") pod \"payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7\" (UID: \"0c2f60ee-cdf2-4a48-a6ff-092f61a349e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7" Oct 30 12:39:13.156971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:13.156947 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7" Oct 30 12:39:13.461814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:13.461785 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7"] Oct 30 12:39:13.462728 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:39:13.462692 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2f60ee_cdf2_4a48_a6ff_092f61a349e2.slice/crio-a5291b5eff714f8632daec0558602e16c6b250de78bd6feff4d50d3ae6164372 WatchSource:0}: Error finding container a5291b5eff714f8632daec0558602e16c6b250de78bd6feff4d50d3ae6164372: Status 404 returned error can't find the container with id a5291b5eff714f8632daec0558602e16c6b250de78bd6feff4d50d3ae6164372 Oct 30 12:39:13.667042 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:13.667013 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7" event={"ID":"0c2f60ee-cdf2-4a48-a6ff-092f61a349e2","Type":"ContainerStarted","Data":"00a38bdedc007d2daf87be0593ffc9d3230e359c97989704e7b277a9d0a89104"} Oct 30 12:39:13.667217 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:13.667065 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7" event={"ID":"0c2f60ee-cdf2-4a48-a6ff-092f61a349e2","Type":"ContainerStarted","Data":"a5291b5eff714f8632daec0558602e16c6b250de78bd6feff4d50d3ae6164372"} Oct 30 12:39:13.667300 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:13.667281 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7" Oct 30 12:39:13.707938 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:13.707883 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7" podStartSLOduration=1.7078697919999999 podStartE2EDuration="1.707869792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:39:13.705828154 +0000 UTC m=+2706.815671054" watchObservedRunningTime="2026-10-30 12:39:13.707869792 +0000 UTC m=+2706.817712672" Oct 30 12:39:13.961588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:13.961542 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:15.512131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:15.512096 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:17.131499 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:17.131469 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:18.750722 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:18.750680 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:20.350888 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:20.350850 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:21.944120 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:21.944084 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:23.503698 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:23.503663 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:24.685114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:24.685080 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7" Oct 30 12:39:25.064491 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:25.064399 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:26.899039 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:26.899005 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:28.526899 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:28.526865 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:30.070985 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:30.070952 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:31.598352 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:31.598317 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:33.218830 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:33.218796 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:33.331133 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:33.331097 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:33.442733 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:33.442689 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:34.966485 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:34.966453 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:35.821760 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:39:35.821724 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:39:36.662543 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:36.662509 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:38.183569 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:38.183534 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:39.729139 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:39.729107 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:41.269541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:41.269506 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:42.982883 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:42.982844 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:44.660071 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:44.660038 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:46.244129 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:46.244095 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:47.787393 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:47.787357 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:49.348868 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:49.348782 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:50.954481 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:50.954447 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:52.656385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:52.656346 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:54.342021 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:54.341989 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:56.048600 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:56.048562 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:57.625669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:57.625640 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:59.313436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:59.313405 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:39:59.424194 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:39:59.424158 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:00.896417 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:00.896384 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:01.782078 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:01.782045 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:02.235353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:02.235315 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:02.569182 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:02.569110 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:02.727752 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:02.727712 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:02.864840 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:02.864751 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:04.285965 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:04.285934 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:05.816060 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:40:05.816024 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:40:05.958414 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:05.958383 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:06.412816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:06.412775 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:06.527025 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:06.526966 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:07.766020 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:07.765983 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:08.126882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.126841 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:08.218389 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.218355 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j"] Oct 30 12:40:08.223055 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.223029 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j" Oct 30 12:40:08.237614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.237579 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-dd0628\"" Oct 30 12:40:08.240141 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.240108 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j"] Oct 30 12:40:08.247593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.247552 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-dd0628-dockercfg-x4gbf\"" Oct 30 12:40:08.263963 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.263910 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:08.273566 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.273513 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k"] Oct 30 12:40:08.278387 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.278340 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k" Oct 30 12:40:08.293782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.293751 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k"] Oct 30 12:40:08.472179 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.472097 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a63529-8b71-40df-b440-d8b5316d112a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k\" (UID: \"d7a63529-8b71-40df-b440-d8b5316d112a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k" Oct 30 12:40:08.479546 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.479506 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b770b1fb-b8b3-4f50-93ea-9e88d2e89922-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j\" (UID: \"b770b1fb-b8b3-4f50-93ea-9e88d2e89922\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j" Oct 30 12:40:08.481307 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.481240 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqzpm\" (UniqueName: \"kubernetes.io/projected/d7a63529-8b71-40df-b440-d8b5316d112a-kube-api-access-bqzpm\") pod \"payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k\" (UID: \"d7a63529-8b71-40df-b440-d8b5316d112a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k" Oct 30 12:40:08.481482 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.481460 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll2q9\" (UniqueName: \"kubernetes.io/projected/b770b1fb-b8b3-4f50-93ea-9e88d2e89922-kube-api-access-ll2q9\") pod \"payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j\" (UID: \"b770b1fb-b8b3-4f50-93ea-9e88d2e89922\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j" Oct 30 12:40:08.583385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.583333 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a63529-8b71-40df-b440-d8b5316d112a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k\" (UID: \"d7a63529-8b71-40df-b440-d8b5316d112a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k" Oct 30 12:40:08.583628 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.583400 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b770b1fb-b8b3-4f50-93ea-9e88d2e89922-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j\" (UID: \"b770b1fb-b8b3-4f50-93ea-9e88d2e89922\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j" Oct 30 12:40:08.583713 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.583692 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqzpm\" (UniqueName: \"kubernetes.io/projected/d7a63529-8b71-40df-b440-d8b5316d112a-kube-api-access-bqzpm\") pod \"payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k\" (UID: \"d7a63529-8b71-40df-b440-d8b5316d112a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k" Oct 30 12:40:08.583824 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.583805 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll2q9\" (UniqueName: \"kubernetes.io/projected/b770b1fb-b8b3-4f50-93ea-9e88d2e89922-kube-api-access-ll2q9\") pod \"payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j\" (UID: \"b770b1fb-b8b3-4f50-93ea-9e88d2e89922\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j" Oct 30 12:40:08.584318 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.584241 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b770b1fb-b8b3-4f50-93ea-9e88d2e89922-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j\" (UID: \"b770b1fb-b8b3-4f50-93ea-9e88d2e89922\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j" Oct 30 12:40:08.584424 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.584340 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a63529-8b71-40df-b440-d8b5316d112a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k\" (UID: \"d7a63529-8b71-40df-b440-d8b5316d112a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k" Oct 30 12:40:08.607058 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.607022 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqzpm\" (UniqueName: \"kubernetes.io/projected/d7a63529-8b71-40df-b440-d8b5316d112a-kube-api-access-bqzpm\") pod \"payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k\" (UID: \"d7a63529-8b71-40df-b440-d8b5316d112a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k" Oct 30 12:40:08.607218 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.607200 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll2q9\" (UniqueName: \"kubernetes.io/projected/b770b1fb-b8b3-4f50-93ea-9e88d2e89922-kube-api-access-ll2q9\") pod \"payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j\" (UID: \"b770b1fb-b8b3-4f50-93ea-9e88d2e89922\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j" Oct 30 12:40:08.839676 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.839575 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j" Oct 30 12:40:08.896953 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:08.896921 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k" Oct 30 12:40:09.148587 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:09.148552 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j"] Oct 30 12:40:09.149062 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:40:09.149030 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb770b1fb_b8b3_4f50_93ea_9e88d2e89922.slice/crio-6fefec6c253125d41727469418a801cef2ec1b72e219d2e75ace4c4282c2f00a WatchSource:0}: Error finding container 6fefec6c253125d41727469418a801cef2ec1b72e219d2e75ace4c4282c2f00a: Status 404 returned error can't find the container with id 6fefec6c253125d41727469418a801cef2ec1b72e219d2e75ace4c4282c2f00a Oct 30 12:40:09.242381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:09.242348 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k"] Oct 30 12:40:09.243134 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:40:09.243102 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a63529_8b71_40df_b440_d8b5316d112a.slice/crio-1ae1497293bc3ec044495c975fef115403cc691c19e323391a4ab1ceca2469f2 WatchSource:0}: Error finding container 1ae1497293bc3ec044495c975fef115403cc691c19e323391a4ab1ceca2469f2: Status 404 returned error can't find the container with id 1ae1497293bc3ec044495c975fef115403cc691c19e323391a4ab1ceca2469f2 Oct 30 12:40:09.442177 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:09.442076 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:10.117291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:10.115427 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j" event={"ID":"b770b1fb-b8b3-4f50-93ea-9e88d2e89922","Type":"ContainerStarted","Data":"69f96fc525f89e08940c8313a54ef0ce60ac157e78204b5641e29e74ba8ddaac"} Oct 30 12:40:10.117291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:10.115518 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j" Oct 30 12:40:10.117291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:10.115544 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j" event={"ID":"b770b1fb-b8b3-4f50-93ea-9e88d2e89922","Type":"ContainerStarted","Data":"6fefec6c253125d41727469418a801cef2ec1b72e219d2e75ace4c4282c2f00a"} Oct 30 12:40:10.117939 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:10.117334 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k" event={"ID":"d7a63529-8b71-40df-b440-d8b5316d112a","Type":"ContainerStarted","Data":"a1a8327bcf340bdb0916af6950688d3a6a585a88abd92f20a0180bc3419f096f"} Oct 30 12:40:10.117939 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:10.117384 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k" event={"ID":"d7a63529-8b71-40df-b440-d8b5316d112a","Type":"ContainerStarted","Data":"1ae1497293bc3ec044495c975fef115403cc691c19e323391a4ab1ceca2469f2"} Oct 30 12:40:10.117939 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:10.117470 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k" Oct 30 12:40:10.157451 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:10.157388 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j" podStartSLOduration=2.157372452 podStartE2EDuration="2.157372452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:10.154803113 +0000 UTC m=+2763.264645994" watchObservedRunningTime="2026-10-30 12:40:10.157372452 +0000 UTC m=+2763.267215380" Oct 30 12:40:10.197651 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:10.197596 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k" podStartSLOduration=2.19757945 podStartE2EDuration="2.19757945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:10.195462737 +0000 UTC m=+2763.305305615" watchObservedRunningTime="2026-10-30 12:40:10.19757945 +0000 UTC m=+2763.307422331" Oct 30 12:40:11.016495 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:11.016454 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:12.873889 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:12.873855 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:14.661957 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:14.661917 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:16.359879 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:16.359851 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:17.951418 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:17.951383 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:19.642912 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:19.642872 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:21.166162 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:21.166133 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j" Oct 30 12:40:21.166604 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:21.166207 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k" Oct 30 12:40:21.293746 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:21.293713 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:22.829156 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:22.829122 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn"] Oct 30 12:40:22.829657 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:22.829489 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" podUID="8b50ec75-3d82-436e-914c-0d667dbeb3ae" containerName="istio-proxy" containerID="cri-o://80e9de8e7a06fcd241091381975114793df6b14db92239ae0ce38819dd78cf7e" gracePeriod=30 Oct 30 12:40:22.867022 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:22.866982 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:24.443378 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:24.443345 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:26.047152 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:26.047112 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:27.617546 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:27.617506 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:28.112120 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.112093 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:40:28.233260 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.233167 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-data\") pod \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " Oct 30 12:40:28.233438 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.233325 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istiod-ca-cert\") pod \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " Oct 30 12:40:28.233438 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.233393 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-workload-socket\") pod \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " Oct 30 12:40:28.233438 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.233413 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-envoy\") pod \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " Oct 30 12:40:28.233604 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.233499 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-data" pod "8b50ec75-3d82-436e-914c-0d667dbeb3ae" (UID: "8b50ec75-3d82-436e-914c-0d667dbeb3ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:28.233604 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.233560 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b50ec75-3d82-436e-914c-0d667dbeb3ae-kube-api-access-z5j74\" (UniqueName: \"kubernetes.io/projected/8b50ec75-3d82-436e-914c-0d667dbeb3ae-kube-api-access-z5j74\") pod \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " Oct 30 12:40:28.233604 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.233579 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-workload-certs\") pod \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " Oct 30 12:40:28.233604 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.233602 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-token\" (UniqueName: \"kubernetes.io/projected/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-token\") pod \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " Oct 30 12:40:28.233826 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.233623 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-credential-socket\") pod \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " Oct 30 12:40:28.233826 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.233652 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-podinfo\") pod \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\" (UID: \"8b50ec75-3d82-436e-914c-0d667dbeb3ae\") " Oct 30 12:40:28.233826 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.233715 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-workload-socket" pod "8b50ec75-3d82-436e-914c-0d667dbeb3ae" (UID: "8b50ec75-3d82-436e-914c-0d667dbeb3ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:28.233826 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.233766 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istiod-ca-cert" pod "8b50ec75-3d82-436e-914c-0d667dbeb3ae" (UID: "8b50ec75-3d82-436e-914c-0d667dbeb3ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:40:28.234027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.233897 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-workload-certs" pod "8b50ec75-3d82-436e-914c-0d667dbeb3ae" (UID: "8b50ec75-3d82-436e-914c-0d667dbeb3ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:28.234027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.233913 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-credential-socket" pod "8b50ec75-3d82-436e-914c-0d667dbeb3ae" (UID: "8b50ec75-3d82-436e-914c-0d667dbeb3ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:28.235058 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.234937 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.235058 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.234966 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.235058 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.234981 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.235058 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.234994 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.235058 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.235008 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.236067 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.236045 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-envoy" pod "8b50ec75-3d82-436e-914c-0d667dbeb3ae" (UID: "8b50ec75-3d82-436e-914c-0d667dbeb3ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:28.236162 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.236135 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-token" pod "8b50ec75-3d82-436e-914c-0d667dbeb3ae" (UID: "8b50ec75-3d82-436e-914c-0d667dbeb3ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:28.236232 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.236219 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-podinfo" pod "8b50ec75-3d82-436e-914c-0d667dbeb3ae" (UID: "8b50ec75-3d82-436e-914c-0d667dbeb3ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:40:28.236332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.236313 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b50ec75-3d82-436e-914c-0d667dbeb3ae-kube-api-access-z5j74" pod "8b50ec75-3d82-436e-914c-0d667dbeb3ae" (UID: "8b50ec75-3d82-436e-914c-0d667dbeb3ae"). InnerVolumeSpecName "kube-api-access-z5j74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:28.320198 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.320166 4413 generic.go:356] "Generic (PLEG): container finished" podID="8b50ec75-3d82-436e-914c-0d667dbeb3ae" containerID="80e9de8e7a06fcd241091381975114793df6b14db92239ae0ce38819dd78cf7e" exitCode=0 Oct 30 12:40:28.320445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.320227 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" Oct 30 12:40:28.320445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.320291 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" event={"ID":"8b50ec75-3d82-436e-914c-0d667dbeb3ae","Type":"ContainerDied","Data":"80e9de8e7a06fcd241091381975114793df6b14db92239ae0ce38819dd78cf7e"} Oct 30 12:40:28.320445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.320367 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn" event={"ID":"8b50ec75-3d82-436e-914c-0d667dbeb3ae","Type":"ContainerDied","Data":"824dd57dab0d0ccf82383cdd35a1e38f667949d606fc4278f4cd58388c36fe5c"} Oct 30 12:40:28.320445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.320401 4413 scope.go:122] "RemoveContainer" containerID="80e9de8e7a06fcd241091381975114793df6b14db92239ae0ce38819dd78cf7e" Oct 30 12:40:28.334085 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.334064 4413 scope.go:122] "RemoveContainer" containerID="80e9de8e7a06fcd241091381975114793df6b14db92239ae0ce38819dd78cf7e" Oct 30 12:40:28.334425 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:40:28.334406 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e9de8e7a06fcd241091381975114793df6b14db92239ae0ce38819dd78cf7e\": container with ID starting with 80e9de8e7a06fcd241091381975114793df6b14db92239ae0ce38819dd78cf7e not found: ID does not exist" containerID="80e9de8e7a06fcd241091381975114793df6b14db92239ae0ce38819dd78cf7e" Oct 30 12:40:28.334497 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.334439 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e9de8e7a06fcd241091381975114793df6b14db92239ae0ce38819dd78cf7e"} err="failed to get container status \"80e9de8e7a06fcd241091381975114793df6b14db92239ae0ce38819dd78cf7e\": rpc error: code = NotFound desc = could not find container \"80e9de8e7a06fcd241091381975114793df6b14db92239ae0ce38819dd78cf7e\": container with ID starting with 80e9de8e7a06fcd241091381975114793df6b14db92239ae0ce38819dd78cf7e not found: ID does not exist" Oct 30 12:40:28.336200 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.336179 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.336317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.336204 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5j74\" (UniqueName: \"kubernetes.io/projected/8b50ec75-3d82-436e-914c-0d667dbeb3ae-kube-api-access-z5j74\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.336317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.336213 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.336317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.336223 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b50ec75-3d82-436e-914c-0d667dbeb3ae-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:28.373039 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.373003 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn"] Oct 30 12:40:28.377202 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.377182 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-4h7hn"] Oct 30 12:40:28.617754 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.617649 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489"] Oct 30 12:40:28.618231 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:28.617914 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" podUID="4847ddfc-5276-42d7-aa4e-33811d0c0dbe" containerName="istio-proxy" containerID="cri-o://87f260a4e4ed66f368af0d4eabea1e6c1e760a15c4afba61b2dd73e1f1409886" gracePeriod=30 Oct 30 12:40:29.024735 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:29.024696 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b50ec75-3d82-436e-914c-0d667dbeb3ae" path="/var/lib/kubelet/pods/8b50ec75-3d82-436e-914c-0d667dbeb3ae/volumes" Oct 30 12:40:29.269969 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:29.269933 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:30.808827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:30.808796 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:30.923069 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:30.923039 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:32.642482 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:32.642449 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:33.899029 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:33.898966 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:40:34.030445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.030410 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-token\" (UniqueName: \"kubernetes.io/projected/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-token\") pod \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " Oct 30 12:40:34.030662 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.030481 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-podinfo\") pod \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " Oct 30 12:40:34.030662 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.030567 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-kube-api-access-zl88q\" (UniqueName: \"kubernetes.io/projected/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-kube-api-access-zl88q\") pod \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " Oct 30 12:40:34.031054 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.031031 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-workload-certs\") pod \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " Oct 30 12:40:34.031166 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.031098 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-data\") pod \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " Oct 30 12:40:34.031166 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.031150 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-credential-socket\") pod \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " Oct 30 12:40:34.031339 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.031185 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istiod-ca-cert\") pod \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " Oct 30 12:40:34.031339 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.031227 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-workload-socket\") pod \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " Oct 30 12:40:34.031339 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.031310 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-envoy\") pod \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\" (UID: \"4847ddfc-5276-42d7-aa4e-33811d0c0dbe\") " Oct 30 12:40:34.031579 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.031513 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-data" pod "4847ddfc-5276-42d7-aa4e-33811d0c0dbe" (UID: "4847ddfc-5276-42d7-aa4e-33811d0c0dbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:34.031579 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.031534 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-credential-socket" pod "4847ddfc-5276-42d7-aa4e-33811d0c0dbe" (UID: "4847ddfc-5276-42d7-aa4e-33811d0c0dbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:34.031688 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.031631 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istiod-ca-cert" pod "4847ddfc-5276-42d7-aa4e-33811d0c0dbe" (UID: "4847ddfc-5276-42d7-aa4e-33811d0c0dbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:40:34.032070 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.031903 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-workload-socket" pod "4847ddfc-5276-42d7-aa4e-33811d0c0dbe" (UID: "4847ddfc-5276-42d7-aa4e-33811d0c0dbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:34.032070 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.032021 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-workload-certs" pod "4847ddfc-5276-42d7-aa4e-33811d0c0dbe" (UID: "4847ddfc-5276-42d7-aa4e-33811d0c0dbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:34.032442 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.032228 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:34.032442 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.032275 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:34.032442 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.032291 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:34.032442 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.032304 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:34.032442 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.032318 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:34.033174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.033146 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-token" pod "4847ddfc-5276-42d7-aa4e-33811d0c0dbe" (UID: "4847ddfc-5276-42d7-aa4e-33811d0c0dbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:34.033360 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.033339 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-podinfo" pod "4847ddfc-5276-42d7-aa4e-33811d0c0dbe" (UID: "4847ddfc-5276-42d7-aa4e-33811d0c0dbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:40:34.033474 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.033456 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-kube-api-access-zl88q" pod "4847ddfc-5276-42d7-aa4e-33811d0c0dbe" (UID: "4847ddfc-5276-42d7-aa4e-33811d0c0dbe"). InnerVolumeSpecName "kube-api-access-zl88q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:34.033841 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.033821 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-envoy" pod "4847ddfc-5276-42d7-aa4e-33811d0c0dbe" (UID: "4847ddfc-5276-42d7-aa4e-33811d0c0dbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:34.133681 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.133642 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:34.133681 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.133680 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:34.133899 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.133691 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl88q\" (UniqueName: \"kubernetes.io/projected/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-kube-api-access-zl88q\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:34.133899 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.133700 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4847ddfc-5276-42d7-aa4e-33811d0c0dbe-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:34.259482 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.259402 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:34.372761 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.372725 4413 generic.go:356] "Generic (PLEG): container finished" podID="4847ddfc-5276-42d7-aa4e-33811d0c0dbe" containerID="87f260a4e4ed66f368af0d4eabea1e6c1e760a15c4afba61b2dd73e1f1409886" exitCode=0 Oct 30 12:40:34.373027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.372779 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" Oct 30 12:40:34.373027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.372794 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" event={"ID":"4847ddfc-5276-42d7-aa4e-33811d0c0dbe","Type":"ContainerDied","Data":"87f260a4e4ed66f368af0d4eabea1e6c1e760a15c4afba61b2dd73e1f1409886"} Oct 30 12:40:34.373027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.372839 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489" event={"ID":"4847ddfc-5276-42d7-aa4e-33811d0c0dbe","Type":"ContainerDied","Data":"bb593adfa34228d27ce56b5d562cd963b1539f9a1df6e18e0d688fd965d9ee05"} Oct 30 12:40:34.373027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.372861 4413 scope.go:122] "RemoveContainer" containerID="87f260a4e4ed66f368af0d4eabea1e6c1e760a15c4afba61b2dd73e1f1409886" Oct 30 12:40:34.384505 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.384487 4413 scope.go:122] "RemoveContainer" containerID="87f260a4e4ed66f368af0d4eabea1e6c1e760a15c4afba61b2dd73e1f1409886" Oct 30 12:40:34.384822 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:40:34.384804 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f260a4e4ed66f368af0d4eabea1e6c1e760a15c4afba61b2dd73e1f1409886\": container with ID starting with 87f260a4e4ed66f368af0d4eabea1e6c1e760a15c4afba61b2dd73e1f1409886 not found: ID does not exist" containerID="87f260a4e4ed66f368af0d4eabea1e6c1e760a15c4afba61b2dd73e1f1409886" Oct 30 12:40:34.384910 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.384830 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f260a4e4ed66f368af0d4eabea1e6c1e760a15c4afba61b2dd73e1f1409886"} err="failed to get container status \"87f260a4e4ed66f368af0d4eabea1e6c1e760a15c4afba61b2dd73e1f1409886\": rpc error: code = NotFound desc = could not find container \"87f260a4e4ed66f368af0d4eabea1e6c1e760a15c4afba61b2dd73e1f1409886\": container with ID starting with 87f260a4e4ed66f368af0d4eabea1e6c1e760a15c4afba61b2dd73e1f1409886 not found: ID does not exist" Oct 30 12:40:34.425627 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.425599 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489"] Oct 30 12:40:34.437087 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:34.437057 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-8k489"] Oct 30 12:40:35.021978 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:35.021942 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4847ddfc-5276-42d7-aa4e-33811d0c0dbe" path="/var/lib/kubelet/pods/4847ddfc-5276-42d7-aa4e-33811d0c0dbe/volumes" Oct 30 12:40:35.815764 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:40:35.815725 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:40:36.048381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:36.048348 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:36.504051 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:36.504020 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q"] Oct 30 12:40:36.504260 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:36.504225 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" podUID="73313e21-a547-4a5d-b1bc-cd154a4b7143" containerName="istio-proxy" containerID="cri-o://c4716799fa34e59bf287bdf168509b2b4bddb6e566891d313197f228207a19e3" gracePeriod=30 Oct 30 12:40:36.733762 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:36.733732 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:37.672945 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:37.672892 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:39.198423 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:39.198383 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:40.892263 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:40.892219 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:41.793024 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.792946 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:40:41.852776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.851374 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-envoy\") pod \"73313e21-a547-4a5d-b1bc-cd154a4b7143\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " Oct 30 12:40:41.852776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.851443 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73313e21-a547-4a5d-b1bc-cd154a4b7143-kube-api-access-wqhg7\" (UniqueName: \"kubernetes.io/projected/73313e21-a547-4a5d-b1bc-cd154a4b7143-kube-api-access-wqhg7\") pod \"73313e21-a547-4a5d-b1bc-cd154a4b7143\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " Oct 30 12:40:41.852776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.851580 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-token\" (UniqueName: \"kubernetes.io/projected/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-token\") pod \"73313e21-a547-4a5d-b1bc-cd154a4b7143\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " Oct 30 12:40:41.852776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.851607 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73313e21-a547-4a5d-b1bc-cd154a4b7143-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73313e21-a547-4a5d-b1bc-cd154a4b7143-istiod-ca-cert\") pod \"73313e21-a547-4a5d-b1bc-cd154a4b7143\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " Oct 30 12:40:41.852776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.851667 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-data\") pod \"73313e21-a547-4a5d-b1bc-cd154a4b7143\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " Oct 30 12:40:41.852776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.851711 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-workload-certs\") pod \"73313e21-a547-4a5d-b1bc-cd154a4b7143\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " Oct 30 12:40:41.852776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.851787 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-podinfo\") pod \"73313e21-a547-4a5d-b1bc-cd154a4b7143\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " Oct 30 12:40:41.852776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.851848 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-credential-socket\") pod \"73313e21-a547-4a5d-b1bc-cd154a4b7143\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " Oct 30 12:40:41.852776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.851922 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-workload-socket\") pod \"73313e21-a547-4a5d-b1bc-cd154a4b7143\" (UID: \"73313e21-a547-4a5d-b1bc-cd154a4b7143\") " Oct 30 12:40:41.852776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.852402 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-data" pod "73313e21-a547-4a5d-b1bc-cd154a4b7143" (UID: "73313e21-a547-4a5d-b1bc-cd154a4b7143"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:41.852776 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.852566 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:41.853718 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.853533 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-workload-certs" pod "73313e21-a547-4a5d-b1bc-cd154a4b7143" (UID: "73313e21-a547-4a5d-b1bc-cd154a4b7143"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:41.853992 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.853877 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-credential-socket" pod "73313e21-a547-4a5d-b1bc-cd154a4b7143" (UID: "73313e21-a547-4a5d-b1bc-cd154a4b7143"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:41.856553 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.856527 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73313e21-a547-4a5d-b1bc-cd154a4b7143-kube-api-access-wqhg7" pod "73313e21-a547-4a5d-b1bc-cd154a4b7143" (UID: "73313e21-a547-4a5d-b1bc-cd154a4b7143"). InnerVolumeSpecName "kube-api-access-wqhg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:41.857366 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.857332 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-workload-socket" pod "73313e21-a547-4a5d-b1bc-cd154a4b7143" (UID: "73313e21-a547-4a5d-b1bc-cd154a4b7143"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:41.857489 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.857365 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-envoy" pod "73313e21-a547-4a5d-b1bc-cd154a4b7143" (UID: "73313e21-a547-4a5d-b1bc-cd154a4b7143"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:40:41.857762 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.857738 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73313e21-a547-4a5d-b1bc-cd154a4b7143-istiod-ca-cert" pod "73313e21-a547-4a5d-b1bc-cd154a4b7143" (UID: "73313e21-a547-4a5d-b1bc-cd154a4b7143"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:40:41.860076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.860041 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-token" pod "73313e21-a547-4a5d-b1bc-cd154a4b7143" (UID: "73313e21-a547-4a5d-b1bc-cd154a4b7143"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:41.871615 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.871586 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-podinfo" pod "73313e21-a547-4a5d-b1bc-cd154a4b7143" (UID: "73313e21-a547-4a5d-b1bc-cd154a4b7143"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:40:41.953695 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.953656 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:41.953695 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.953693 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:41.953695 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.953703 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:41.954149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.953712 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqhg7\" (UniqueName: \"kubernetes.io/projected/73313e21-a547-4a5d-b1bc-cd154a4b7143-kube-api-access-wqhg7\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:41.954149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.953721 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:41.954149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.953729 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73313e21-a547-4a5d-b1bc-cd154a4b7143-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:41.954149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.953736 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73313e21-a547-4a5d-b1bc-cd154a4b7143-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:41.954149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:41.953743 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73313e21-a547-4a5d-b1bc-cd154a4b7143-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:42.430303 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:42.430265 4413 generic.go:356] "Generic (PLEG): container finished" podID="73313e21-a547-4a5d-b1bc-cd154a4b7143" containerID="c4716799fa34e59bf287bdf168509b2b4bddb6e566891d313197f228207a19e3" exitCode=0 Oct 30 12:40:42.430495 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:42.430341 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" Oct 30 12:40:42.430495 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:42.430346 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" event={"ID":"73313e21-a547-4a5d-b1bc-cd154a4b7143","Type":"ContainerDied","Data":"c4716799fa34e59bf287bdf168509b2b4bddb6e566891d313197f228207a19e3"} Oct 30 12:40:42.430495 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:42.430402 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q" event={"ID":"73313e21-a547-4a5d-b1bc-cd154a4b7143","Type":"ContainerDied","Data":"7eb1214d120fb8a5ffd895cbf722a8fb3f027cf4359bc8020e1c7a618211fa88"} Oct 30 12:40:42.430495 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:42.430426 4413 scope.go:122] "RemoveContainer" containerID="c4716799fa34e59bf287bdf168509b2b4bddb6e566891d313197f228207a19e3" Oct 30 12:40:42.443394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:42.443377 4413 scope.go:122] "RemoveContainer" containerID="c4716799fa34e59bf287bdf168509b2b4bddb6e566891d313197f228207a19e3" Oct 30 12:40:42.443677 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:40:42.443660 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4716799fa34e59bf287bdf168509b2b4bddb6e566891d313197f228207a19e3\": container with ID starting with c4716799fa34e59bf287bdf168509b2b4bddb6e566891d313197f228207a19e3 not found: ID does not exist" containerID="c4716799fa34e59bf287bdf168509b2b4bddb6e566891d313197f228207a19e3" Oct 30 12:40:42.443741 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:42.443690 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4716799fa34e59bf287bdf168509b2b4bddb6e566891d313197f228207a19e3"} err="failed to get container status \"c4716799fa34e59bf287bdf168509b2b4bddb6e566891d313197f228207a19e3\": rpc error: code = NotFound desc = could not find container \"c4716799fa34e59bf287bdf168509b2b4bddb6e566891d313197f228207a19e3\": container with ID starting with c4716799fa34e59bf287bdf168509b2b4bddb6e566891d313197f228207a19e3 not found: ID does not exist" Oct 30 12:40:42.459513 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:42.459489 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:42.483040 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:42.483009 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q"] Oct 30 12:40:42.495032 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:42.494997 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-9ld6q"] Oct 30 12:40:42.716596 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:42.716519 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:43.022857 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:43.022777 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73313e21-a547-4a5d-b1bc-cd154a4b7143" path="/var/lib/kubelet/pods/73313e21-a547-4a5d-b1bc-cd154a4b7143/volumes" Oct 30 12:40:44.166894 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:44.166857 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:45.696493 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:45.696449 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:47.266297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.266214 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:47.554076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.553925 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:47.656350 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.656233 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11581-ccb939"] Oct 30 12:40:47.657385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.657360 4413 cpu_manager.go:414] "containerMap: removing container" podUID="8b50ec75-3d82-436e-914c-0d667dbeb3ae" containerName="istio-proxy" Oct 30 12:40:47.657385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.657384 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b50ec75-3d82-436e-914c-0d667dbeb3ae" containerName="istio-proxy" Oct 30 12:40:47.657579 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.657399 4413 cpu_manager.go:414] "containerMap: removing container" podUID="73313e21-a547-4a5d-b1bc-cd154a4b7143" containerName="istio-proxy" Oct 30 12:40:47.657579 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.657406 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73313e21-a547-4a5d-b1bc-cd154a4b7143" containerName="istio-proxy" Oct 30 12:40:47.657579 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.657420 4413 cpu_manager.go:414] "containerMap: removing container" podUID="4847ddfc-5276-42d7-aa4e-33811d0c0dbe" containerName="istio-proxy" Oct 30 12:40:47.657579 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.657427 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4847ddfc-5276-42d7-aa4e-33811d0c0dbe" containerName="istio-proxy" Oct 30 12:40:47.657795 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.657613 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b50ec75-3d82-436e-914c-0d667dbeb3ae" containerName="istio-proxy" Oct 30 12:40:47.657795 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.657622 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="73313e21-a547-4a5d-b1bc-cd154a4b7143" containerName="istio-proxy" Oct 30 12:40:47.657795 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.657630 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="4847ddfc-5276-42d7-aa4e-33811d0c0dbe" containerName="istio-proxy" Oct 30 12:40:47.663797 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.663771 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-ccb939" Oct 30 12:40:47.675043 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.675008 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11581-ccb939"] Oct 30 12:40:47.682722 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.682696 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ttfhc\"" Oct 30 12:40:47.839084 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.839002 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txhth\" (UniqueName: \"kubernetes.io/projected/79f33fac-0548-4c4b-b471-8a4e6b664e0e-kube-api-access-txhth\") pod \"test-curl-11581-ccb939\" (UID: \"79f33fac-0548-4c4b-b471-8a4e6b664e0e\") " pod="opendatahub/test-curl-11581-ccb939" Oct 30 12:40:47.940101 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.940063 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txhth\" (UniqueName: \"kubernetes.io/projected/79f33fac-0548-4c4b-b471-8a4e6b664e0e-kube-api-access-txhth\") pod \"test-curl-11581-ccb939\" (UID: \"79f33fac-0548-4c4b-b471-8a4e6b664e0e\") " pod="opendatahub/test-curl-11581-ccb939" Oct 30 12:40:47.963088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.963061 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txhth\" (UniqueName: \"kubernetes.io/projected/79f33fac-0548-4c4b-b471-8a4e6b664e0e-kube-api-access-txhth\") pod \"test-curl-11581-ccb939\" (UID: \"79f33fac-0548-4c4b-b471-8a4e6b664e0e\") " pod="opendatahub/test-curl-11581-ccb939" Oct 30 12:40:47.986971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:47.986935 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-ccb939" Oct 30 12:40:48.290910 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:48.290885 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11581-ccb939"] Oct 30 12:40:48.292660 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:40:48.292635 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79f33fac_0548_4c4b_b471_8a4e6b664e0e.slice/crio-89df4663083b004751a7b730d7b34f99839009f6f428bca7733f6047e19317f6 WatchSource:0}: Error finding container 89df4663083b004751a7b730d7b34f99839009f6f428bca7733f6047e19317f6: Status 404 returned error can't find the container with id 89df4663083b004751a7b730d7b34f99839009f6f428bca7733f6047e19317f6 Oct 30 12:40:48.294828 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:48.294806 4413 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:40:48.487285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:48.487242 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11581-ccb939" event={"ID":"79f33fac-0548-4c4b-b471-8a4e6b664e0e","Type":"ContainerStarted","Data":"89df4663083b004751a7b730d7b34f99839009f6f428bca7733f6047e19317f6"} Oct 30 12:40:48.949813 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:48.949760 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:49.497121 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:49.497085 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11581-ccb939" event={"ID":"79f33fac-0548-4c4b-b471-8a4e6b664e0e","Type":"ContainerStarted","Data":"e78fcd418ca07053840d11fccf9082ecf54793836b1e94abc623519858bf0693"} Oct 30 12:40:49.534107 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:49.534051 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11581-ccb939" podStartSLOduration=1.857899242 podStartE2EDuration="2.534038303s" totalImagesPullingTime="676.139061ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:49.53354949 +0000 UTC m=+2802.643392374" watchObservedRunningTime="2026-10-30 12:40:49.534038303 +0000 UTC m=+2802.643881183" Oct 30 12:40:50.454126 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:50.453213 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11581-ccb939"] Oct 30 12:40:50.457026 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:50.457001 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11581-ccb939"] Oct 30 12:40:50.541028 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:50.540996 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:50.610639 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:50.610606 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11581-beec74"] Oct 30 12:40:50.611431 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:50.611405 4413 cpu_manager.go:414] "containerMap: removing container" podUID="79f33fac-0548-4c4b-b471-8a4e6b664e0e" containerName="test-curl-11581-ccb939" Oct 30 12:40:50.611431 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:50.611431 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f33fac-0548-4c4b-b471-8a4e6b664e0e" containerName="test-curl-11581-ccb939" Oct 30 12:40:50.611693 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:50.611677 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f33fac-0548-4c4b-b471-8a4e6b664e0e" containerName="test-curl-11581-ccb939" Oct 30 12:40:50.615575 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:50.615556 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-beec74" Oct 30 12:40:50.628490 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:50.628468 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11581-beec74"] Oct 30 12:40:50.779625 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:50.779528 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5ddr\" (UniqueName: \"kubernetes.io/projected/e2f18816-1ab7-4462-8800-e8e628002df0-kube-api-access-j5ddr\") pod \"test-curl-11581-beec74\" (UID: \"e2f18816-1ab7-4462-8800-e8e628002df0\") " pod="opendatahub/test-curl-11581-beec74" Oct 30 12:40:50.881367 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:50.881334 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5ddr\" (UniqueName: \"kubernetes.io/projected/e2f18816-1ab7-4462-8800-e8e628002df0-kube-api-access-j5ddr\") pod \"test-curl-11581-beec74\" (UID: \"e2f18816-1ab7-4462-8800-e8e628002df0\") " pod="opendatahub/test-curl-11581-beec74" Oct 30 12:40:50.904829 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:50.904793 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5ddr\" (UniqueName: \"kubernetes.io/projected/e2f18816-1ab7-4462-8800-e8e628002df0-kube-api-access-j5ddr\") pod \"test-curl-11581-beec74\" (UID: \"e2f18816-1ab7-4462-8800-e8e628002df0\") " pod="opendatahub/test-curl-11581-beec74" Oct 30 12:40:50.930986 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:50.930958 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-beec74" Oct 30 12:40:51.237079 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:51.233014 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11581-beec74"] Oct 30 12:40:51.513208 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:51.513117 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11581-beec74" event={"ID":"e2f18816-1ab7-4462-8800-e8e628002df0","Type":"ContainerStarted","Data":"c8f445538e60262038172735da8a4d844ebbff692b32f05a60a79445289dd824"} Oct 30 12:40:51.513208 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:51.513166 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11581-beec74" event={"ID":"e2f18816-1ab7-4462-8800-e8e628002df0","Type":"ContainerStarted","Data":"b945d468a1ec95fe0b5ce93f9d68ac27d0f4c128142e35e3acee88f9c0384245"} Oct 30 12:40:51.513487 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:51.513214 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11581-ccb939" podUID="79f33fac-0548-4c4b-b471-8a4e6b664e0e" containerName="test-curl-11581-ccb939" containerID="cri-o://e78fcd418ca07053840d11fccf9082ecf54793836b1e94abc623519858bf0693" gracePeriod=2 Oct 30 12:40:51.553759 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:51.553703 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11581-beec74" podStartSLOduration=1.553688253 podStartE2EDuration="1.553688253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:40:51.550436987 +0000 UTC m=+2804.660279868" watchObservedRunningTime="2026-10-30 12:40:51.553688253 +0000 UTC m=+2804.663531133" Oct 30 12:40:52.134460 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:52.134430 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:52.565415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:52.565238 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11581-beec74"] Oct 30 12:40:52.569428 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:52.569392 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11581-beec74"] Oct 30 12:40:53.531690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:53.531646 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11581-beec74" podUID="e2f18816-1ab7-4462-8800-e8e628002df0" containerName="test-curl-11581-beec74" containerID="cri-o://c8f445538e60262038172735da8a4d844ebbff692b32f05a60a79445289dd824" gracePeriod=2 Oct 30 12:40:53.693929 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:53.693904 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-ccb939" Oct 30 12:40:53.771385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:53.771287 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f33fac-0548-4c4b-b471-8a4e6b664e0e-kube-api-access-txhth\" (UniqueName: \"kubernetes.io/projected/79f33fac-0548-4c4b-b471-8a4e6b664e0e-kube-api-access-txhth\") pod \"79f33fac-0548-4c4b-b471-8a4e6b664e0e\" (UID: \"79f33fac-0548-4c4b-b471-8a4e6b664e0e\") " Oct 30 12:40:53.773750 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:53.773715 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f33fac-0548-4c4b-b471-8a4e6b664e0e-kube-api-access-txhth" pod "79f33fac-0548-4c4b-b471-8a4e6b664e0e" (UID: "79f33fac-0548-4c4b-b471-8a4e6b664e0e"). InnerVolumeSpecName "kube-api-access-txhth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:53.873191 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:53.873158 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txhth\" (UniqueName: \"kubernetes.io/projected/79f33fac-0548-4c4b-b471-8a4e6b664e0e-kube-api-access-txhth\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:54.028553 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:54.028471 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:54.541020 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:54.540987 4413 generic.go:356] "Generic (PLEG): container finished" podID="79f33fac-0548-4c4b-b471-8a4e6b664e0e" containerID="e78fcd418ca07053840d11fccf9082ecf54793836b1e94abc623519858bf0693" exitCode=137 Oct 30 12:40:54.541306 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:54.541039 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-ccb939" Oct 30 12:40:54.541306 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:54.541089 4413 scope.go:122] "RemoveContainer" containerID="e78fcd418ca07053840d11fccf9082ecf54793836b1e94abc623519858bf0693" Oct 30 12:40:54.552866 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:54.552849 4413 scope.go:122] "RemoveContainer" containerID="e78fcd418ca07053840d11fccf9082ecf54793836b1e94abc623519858bf0693" Oct 30 12:40:54.553146 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:40:54.553129 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e78fcd418ca07053840d11fccf9082ecf54793836b1e94abc623519858bf0693\": container with ID starting with e78fcd418ca07053840d11fccf9082ecf54793836b1e94abc623519858bf0693 not found: ID does not exist" containerID="e78fcd418ca07053840d11fccf9082ecf54793836b1e94abc623519858bf0693" Oct 30 12:40:54.553186 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:54.553155 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e78fcd418ca07053840d11fccf9082ecf54793836b1e94abc623519858bf0693"} err="failed to get container status \"e78fcd418ca07053840d11fccf9082ecf54793836b1e94abc623519858bf0693\": rpc error: code = NotFound desc = could not find container \"e78fcd418ca07053840d11fccf9082ecf54793836b1e94abc623519858bf0693\": container with ID starting with e78fcd418ca07053840d11fccf9082ecf54793836b1e94abc623519858bf0693 not found: ID does not exist" Oct 30 12:40:54.566201 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:40:54.566174 4413 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11581-ccb939\" is forbidden: User \"system:node:ip-10-0-138-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-23.ec2.internal' and this object" podUID="79f33fac-0548-4c4b-b471-8a4e6b664e0e" pod="opendatahub/test-curl-11581-ccb939" Oct 30 12:40:55.022016 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:55.021983 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f33fac-0548-4c4b-b471-8a4e6b664e0e" path="/var/lib/kubelet/pods/79f33fac-0548-4c4b-b471-8a4e6b664e0e/volumes" Oct 30 12:40:55.770614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:55.770586 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-beec74" Oct 30 12:40:55.800309 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:55.800185 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f18816-1ab7-4462-8800-e8e628002df0-kube-api-access-j5ddr\" (UniqueName: \"kubernetes.io/projected/e2f18816-1ab7-4462-8800-e8e628002df0-kube-api-access-j5ddr\") pod \"e2f18816-1ab7-4462-8800-e8e628002df0\" (UID: \"e2f18816-1ab7-4462-8800-e8e628002df0\") " Oct 30 12:40:55.802535 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:55.802511 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f18816-1ab7-4462-8800-e8e628002df0-kube-api-access-j5ddr" pod "e2f18816-1ab7-4462-8800-e8e628002df0" (UID: "e2f18816-1ab7-4462-8800-e8e628002df0"). InnerVolumeSpecName "kube-api-access-j5ddr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:40:55.836138 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:55.836111 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:55.901838 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:55.901806 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5ddr\" (UniqueName: \"kubernetes.io/projected/e2f18816-1ab7-4462-8800-e8e628002df0-kube-api-access-j5ddr\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:40:56.558241 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:56.558203 4413 generic.go:356] "Generic (PLEG): container finished" podID="e2f18816-1ab7-4462-8800-e8e628002df0" containerID="c8f445538e60262038172735da8a4d844ebbff692b32f05a60a79445289dd824" exitCode=137 Oct 30 12:40:56.558756 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:56.558300 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11581-beec74" Oct 30 12:40:56.558756 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:56.558321 4413 scope.go:122] "RemoveContainer" containerID="c8f445538e60262038172735da8a4d844ebbff692b32f05a60a79445289dd824" Oct 30 12:40:56.568354 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:56.568337 4413 scope.go:122] "RemoveContainer" containerID="c8f445538e60262038172735da8a4d844ebbff692b32f05a60a79445289dd824" Oct 30 12:40:56.568621 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:40:56.568605 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f445538e60262038172735da8a4d844ebbff692b32f05a60a79445289dd824\": container with ID starting with c8f445538e60262038172735da8a4d844ebbff692b32f05a60a79445289dd824 not found: ID does not exist" containerID="c8f445538e60262038172735da8a4d844ebbff692b32f05a60a79445289dd824" Oct 30 12:40:56.568687 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:56.568629 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f445538e60262038172735da8a4d844ebbff692b32f05a60a79445289dd824"} err="failed to get container status \"c8f445538e60262038172735da8a4d844ebbff692b32f05a60a79445289dd824\": rpc error: code = NotFound desc = could not find container \"c8f445538e60262038172735da8a4d844ebbff692b32f05a60a79445289dd824\": container with ID starting with c8f445538e60262038172735da8a4d844ebbff692b32f05a60a79445289dd824 not found: ID does not exist" Oct 30 12:40:56.579851 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:40:56.579823 4413 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11581-beec74\" is forbidden: User \"system:node:ip-10-0-138-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-23.ec2.internal' and this object" podUID="e2f18816-1ab7-4462-8800-e8e628002df0" pod="opendatahub/test-curl-11581-beec74" Oct 30 12:40:57.032558 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:57.032516 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f18816-1ab7-4462-8800-e8e628002df0" path="/var/lib/kubelet/pods/e2f18816-1ab7-4462-8800-e8e628002df0/volumes" Oct 30 12:40:57.039965 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:40:57.039928 4413 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11581-beec74\" is forbidden: User \"system:node:ip-10-0-138-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-23.ec2.internal' and this object" podUID="e2f18816-1ab7-4462-8800-e8e628002df0" pod="opendatahub/test-curl-11581-beec74" Oct 30 12:40:57.650895 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:57.650861 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:59.440382 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:59.440354 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:40:59.569217 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:40:59.569187 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:00.163105 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:00.163070 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:00.674010 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:00.673973 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:00.949187 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:00.949107 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:01.362768 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:01.362688 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:02.560243 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:02.560178 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:02.917830 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:02.917780 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8"] Oct 30 12:41:02.963107 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:02.963071 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:03.376694 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:03.376664 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:04.089732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:04.089692 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:04.202609 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:04.202565 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:04.576730 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:04.576697 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:04.941175 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:04.941142 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:05.086043 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:05.086013 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:05.262333 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:05.260655 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:05.483973 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:05.483937 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:05.817468 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:41:05.817424 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:41:05.909372 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:05.909336 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:06.151527 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:06.151487 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:07.176512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:07.176474 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:08.926031 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:08.925995 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:09.046750 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:09.046720 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:09.568098 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:09.568059 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:09.701380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:09.701328 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:10.673836 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:10.673785 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:10.808421 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:10.808389 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:10.996357 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:10.996218 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:12.312354 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:12.312311 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:12.450071 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:12.450037 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:12.619744 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:12.619651 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:14.421883 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:14.421841 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:16.064965 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:16.064933 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:17.850313 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:17.850227 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:17.944634 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:17.944590 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" podUID="676182ad-449e-49da-822a-a4af9861f76b" containerName="main" containerID="cri-o://b34f8762e6b88323b2367c64b6b53e8563d8e8455d01436205cf8efcb496dd38" gracePeriod=45 Oct 30 12:41:18.221882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.221852 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:41:18.304112 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.304081 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-model-cache\") pod \"676182ad-449e-49da-822a-a4af9861f76b\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " Oct 30 12:41:18.304341 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.304146 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-tmp-dir\") pod \"676182ad-449e-49da-822a-a4af9861f76b\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " Oct 30 12:41:18.304341 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.304197 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/676182ad-449e-49da-822a-a4af9861f76b-tls-certs\" (UniqueName: \"kubernetes.io/secret/676182ad-449e-49da-822a-a4af9861f76b-tls-certs\") pod \"676182ad-449e-49da-822a-a4af9861f76b\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " Oct 30 12:41:18.304341 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.304303 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676182ad-449e-49da-822a-a4af9861f76b-kube-api-access-kvddz\" (UniqueName: \"kubernetes.io/projected/676182ad-449e-49da-822a-a4af9861f76b-kube-api-access-kvddz\") pod \"676182ad-449e-49da-822a-a4af9861f76b\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " Oct 30 12:41:18.304341 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.304340 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-dshm\") pod \"676182ad-449e-49da-822a-a4af9861f76b\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " Oct 30 12:41:18.304515 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.304442 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-model-cache" pod "676182ad-449e-49da-822a-a4af9861f76b" (UID: "676182ad-449e-49da-822a-a4af9861f76b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:18.304515 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.304452 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-tmp-dir" pod "676182ad-449e-49da-822a-a4af9861f76b" (UID: "676182ad-449e-49da-822a-a4af9861f76b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:18.304596 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.304521 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-home\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-home\") pod \"676182ad-449e-49da-822a-a4af9861f76b\" (UID: \"676182ad-449e-49da-822a-a4af9861f76b\") " Oct 30 12:41:18.304812 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.304790 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-home" pod "676182ad-449e-49da-822a-a4af9861f76b" (UID: "676182ad-449e-49da-822a-a4af9861f76b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:18.305198 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.305181 4413 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:18.305302 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.305203 4413 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:18.305302 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.305217 4413 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:18.306790 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.306768 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-dshm" pod "676182ad-449e-49da-822a-a4af9861f76b" (UID: "676182ad-449e-49da-822a-a4af9861f76b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:18.307115 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.307095 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/676182ad-449e-49da-822a-a4af9861f76b-tls-certs" pod "676182ad-449e-49da-822a-a4af9861f76b" (UID: "676182ad-449e-49da-822a-a4af9861f76b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:41:18.307115 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.307104 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676182ad-449e-49da-822a-a4af9861f76b-kube-api-access-kvddz" pod "676182ad-449e-49da-822a-a4af9861f76b" (UID: "676182ad-449e-49da-822a-a4af9861f76b"). InnerVolumeSpecName "kube-api-access-kvddz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:18.407141 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.407108 4413 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/676182ad-449e-49da-822a-a4af9861f76b-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:18.407141 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.407140 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvddz\" (UniqueName: \"kubernetes.io/projected/676182ad-449e-49da-822a-a4af9861f76b-kube-api-access-kvddz\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:18.407358 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.407150 4413 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/676182ad-449e-49da-822a-a4af9861f76b-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:18.762612 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.762531 4413 generic.go:356] "Generic (PLEG): container finished" podID="676182ad-449e-49da-822a-a4af9861f76b" containerID="b34f8762e6b88323b2367c64b6b53e8563d8e8455d01436205cf8efcb496dd38" exitCode=0 Oct 30 12:41:18.762612 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.762583 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" Oct 30 12:41:18.762801 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.762590 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" event={"ID":"676182ad-449e-49da-822a-a4af9861f76b","Type":"ContainerDied","Data":"b34f8762e6b88323b2367c64b6b53e8563d8e8455d01436205cf8efcb496dd38"} Oct 30 12:41:18.762801 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.762687 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8" event={"ID":"676182ad-449e-49da-822a-a4af9861f76b","Type":"ContainerDied","Data":"b45a56a492a71d32e58dd106917b962bf21eb417117c35cc816cb6e2f1794a38"} Oct 30 12:41:18.762801 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.762706 4413 scope.go:122] "RemoveContainer" containerID="b34f8762e6b88323b2367c64b6b53e8563d8e8455d01436205cf8efcb496dd38" Oct 30 12:41:18.774340 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.774318 4413 scope.go:122] "RemoveContainer" containerID="b34f8762e6b88323b2367c64b6b53e8563d8e8455d01436205cf8efcb496dd38" Oct 30 12:41:18.774626 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:41:18.774605 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34f8762e6b88323b2367c64b6b53e8563d8e8455d01436205cf8efcb496dd38\": container with ID starting with b34f8762e6b88323b2367c64b6b53e8563d8e8455d01436205cf8efcb496dd38 not found: ID does not exist" containerID="b34f8762e6b88323b2367c64b6b53e8563d8e8455d01436205cf8efcb496dd38" Oct 30 12:41:18.774700 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.774634 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34f8762e6b88323b2367c64b6b53e8563d8e8455d01436205cf8efcb496dd38"} err="failed to get container status \"b34f8762e6b88323b2367c64b6b53e8563d8e8455d01436205cf8efcb496dd38\": rpc error: code = NotFound desc = could not find container \"b34f8762e6b88323b2367c64b6b53e8563d8e8455d01436205cf8efcb496dd38\": container with ID starting with b34f8762e6b88323b2367c64b6b53e8563d8e8455d01436205cf8efcb496dd38 not found: ID does not exist" Oct 30 12:41:18.815209 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.815172 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8"] Oct 30 12:41:18.827613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:18.827583 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9d45fb/auth-test-model-9d45fb-kserve-687dcffb4f-zxmh8"] Oct 30 12:41:19.021778 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:19.021678 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676182ad-449e-49da-822a-a4af9861f76b" path="/var/lib/kubelet/pods/676182ad-449e-49da-822a-a4af9861f76b/volumes" Oct 30 12:41:19.551681 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:19.551635 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:21.093689 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:21.093649 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:21.648772 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:21.648740 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp"] Oct 30 12:41:21.648993 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:21.648956 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" podUID="fdd40382-d780-4cab-941f-b2ef96ac86a0" containerName="istio-proxy" containerID="cri-o://fb4e7cd896b2f50d531fe82924ebfbe19b6aa45cac14c0d92e2cc2ea1334a713" gracePeriod=30 Oct 30 12:41:21.884949 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:21.884914 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:22.950825 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:22.950789 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:23.422124 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:23.422085 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:24.077500 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:24.077469 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:24.199089 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:24.199057 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:24.654526 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:24.654473 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:26.189610 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:26.189577 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf"] Oct 30 12:41:26.235603 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:26.235573 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:26.346741 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:26.346703 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:26.824761 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:26.824728 4413 generic.go:356] "Generic (PLEG): container finished" podID="fdd40382-d780-4cab-941f-b2ef96ac86a0" containerID="fb4e7cd896b2f50d531fe82924ebfbe19b6aa45cac14c0d92e2cc2ea1334a713" exitCode=0 Oct 30 12:41:26.824935 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:26.824803 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" event={"ID":"fdd40382-d780-4cab-941f-b2ef96ac86a0","Type":"ContainerDied","Data":"fb4e7cd896b2f50d531fe82924ebfbe19b6aa45cac14c0d92e2cc2ea1334a713"} Oct 30 12:41:26.924052 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:26.923981 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:41:27.059594 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.059561 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-credential-socket\") pod \"fdd40382-d780-4cab-941f-b2ef96ac86a0\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " Oct 30 12:41:27.059805 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.059611 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-podinfo\") pod \"fdd40382-d780-4cab-941f-b2ef96ac86a0\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " Oct 30 12:41:27.059805 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.059661 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-token\" (UniqueName: \"kubernetes.io/projected/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-token\") pod \"fdd40382-d780-4cab-941f-b2ef96ac86a0\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " Oct 30 12:41:27.059805 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.059707 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-workload-certs\") pod \"fdd40382-d780-4cab-941f-b2ef96ac86a0\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " Oct 30 12:41:27.059973 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.059851 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-workload-socket\") pod \"fdd40382-d780-4cab-941f-b2ef96ac86a0\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " Oct 30 12:41:27.059973 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.059873 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-data\") pod \"fdd40382-d780-4cab-941f-b2ef96ac86a0\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " Oct 30 12:41:27.059973 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.059876 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-credential-socket" pod "fdd40382-d780-4cab-941f-b2ef96ac86a0" (UID: "fdd40382-d780-4cab-941f-b2ef96ac86a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:27.059973 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.059899 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd40382-d780-4cab-941f-b2ef96ac86a0-kube-api-access-9jb5j\" (UniqueName: \"kubernetes.io/projected/fdd40382-d780-4cab-941f-b2ef96ac86a0-kube-api-access-9jb5j\") pod \"fdd40382-d780-4cab-941f-b2ef96ac86a0\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " Oct 30 12:41:27.059973 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.059923 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-envoy\") pod \"fdd40382-d780-4cab-941f-b2ef96ac86a0\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " Oct 30 12:41:27.060273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.059978 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdd40382-d780-4cab-941f-b2ef96ac86a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd40382-d780-4cab-941f-b2ef96ac86a0-istiod-ca-cert\") pod \"fdd40382-d780-4cab-941f-b2ef96ac86a0\" (UID: \"fdd40382-d780-4cab-941f-b2ef96ac86a0\") " Oct 30 12:41:27.060273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.060055 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-workload-certs" pod "fdd40382-d780-4cab-941f-b2ef96ac86a0" (UID: "fdd40382-d780-4cab-941f-b2ef96ac86a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:27.060273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.060093 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-workload-socket" pod "fdd40382-d780-4cab-941f-b2ef96ac86a0" (UID: "fdd40382-d780-4cab-941f-b2ef96ac86a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:27.060273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.060130 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-data" pod "fdd40382-d780-4cab-941f-b2ef96ac86a0" (UID: "fdd40382-d780-4cab-941f-b2ef96ac86a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:27.060667 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.060486 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd40382-d780-4cab-941f-b2ef96ac86a0-istiod-ca-cert" pod "fdd40382-d780-4cab-941f-b2ef96ac86a0" (UID: "fdd40382-d780-4cab-941f-b2ef96ac86a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:41:27.060879 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.060855 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:27.060954 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.060884 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:27.060954 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.060898 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdd40382-d780-4cab-941f-b2ef96ac86a0-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:27.060954 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.060912 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:27.060954 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.060925 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:27.062236 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.062207 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-podinfo" pod "fdd40382-d780-4cab-941f-b2ef96ac86a0" (UID: "fdd40382-d780-4cab-941f-b2ef96ac86a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:41:27.062236 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.062242 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd40382-d780-4cab-941f-b2ef96ac86a0-kube-api-access-9jb5j" pod "fdd40382-d780-4cab-941f-b2ef96ac86a0" (UID: "fdd40382-d780-4cab-941f-b2ef96ac86a0"). InnerVolumeSpecName "kube-api-access-9jb5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:27.062550 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.062531 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-token" pod "fdd40382-d780-4cab-941f-b2ef96ac86a0" (UID: "fdd40382-d780-4cab-941f-b2ef96ac86a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:27.062677 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.062664 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-envoy" pod "fdd40382-d780-4cab-941f-b2ef96ac86a0" (UID: "fdd40382-d780-4cab-941f-b2ef96ac86a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:27.162687 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.162651 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:27.162687 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.162678 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:27.162687 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.162687 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jb5j\" (UniqueName: \"kubernetes.io/projected/fdd40382-d780-4cab-941f-b2ef96ac86a0-kube-api-access-9jb5j\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:27.162687 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.162696 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdd40382-d780-4cab-941f-b2ef96ac86a0-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:27.833807 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.833781 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" Oct 30 12:41:27.834330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.833774 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp" event={"ID":"fdd40382-d780-4cab-941f-b2ef96ac86a0","Type":"ContainerDied","Data":"d45926f41cb448514425256f23ca46400cd395fcd267c417a53874d24c824de4"} Oct 30 12:41:27.834330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.833925 4413 scope.go:122] "RemoveContainer" containerID="fb4e7cd896b2f50d531fe82924ebfbe19b6aa45cac14c0d92e2cc2ea1334a713" Oct 30 12:41:27.899868 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.899837 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp"] Oct 30 12:41:27.903621 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.903591 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:27.904667 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:27.904649 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-npddp"] Oct 30 12:41:28.013696 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:28.013658 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:28.406467 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:28.406432 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:28.632637 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:28.632578 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:29.021588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:29.021558 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdd40382-d780-4cab-941f-b2ef96ac86a0" path="/var/lib/kubelet/pods/fdd40382-d780-4cab-941f-b2ef96ac86a0/volumes" Oct 30 12:41:29.657280 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:29.657214 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:31.331358 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.331323 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk"] Oct 30 12:41:31.332428 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.332400 4413 cpu_manager.go:414] "containerMap: removing container" podUID="fdd40382-d780-4cab-941f-b2ef96ac86a0" containerName="istio-proxy" Oct 30 12:41:31.332428 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.332423 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdd40382-d780-4cab-941f-b2ef96ac86a0" containerName="istio-proxy" Oct 30 12:41:31.332525 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.332474 4413 cpu_manager.go:414] "containerMap: removing container" podUID="676182ad-449e-49da-822a-a4af9861f76b" containerName="main" Oct 30 12:41:31.332525 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.332481 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676182ad-449e-49da-822a-a4af9861f76b" containerName="main" Oct 30 12:41:31.332525 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.332501 4413 cpu_manager.go:414] "containerMap: removing container" podUID="e2f18816-1ab7-4462-8800-e8e628002df0" containerName="test-curl-11581-beec74" Oct 30 12:41:31.332525 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.332506 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f18816-1ab7-4462-8800-e8e628002df0" containerName="test-curl-11581-beec74" Oct 30 12:41:31.332658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.332615 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="676182ad-449e-49da-822a-a4af9861f76b" containerName="main" Oct 30 12:41:31.332658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.332628 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f18816-1ab7-4462-8800-e8e628002df0" containerName="test-curl-11581-beec74" Oct 30 12:41:31.332658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.332640 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdd40382-d780-4cab-941f-b2ef96ac86a0" containerName="istio-proxy" Oct 30 12:41:31.337861 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.337842 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 12:41:31.363891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.363854 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk"] Oct 30 12:41:31.366049 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.366024 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-a786af-serving-cert\"" Oct 30 12:41:31.421783 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.421749 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:31.434362 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.434330 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk\" (UID: \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 12:41:31.434578 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.434553 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk\" (UID: \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 12:41:31.434630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.434595 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-metrics-certs\") pod \"maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk\" (UID: \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 12:41:31.434691 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.434674 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptj7s\" (UniqueName: \"kubernetes.io/projected/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-kube-api-access-ptj7s\") pod \"maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk\" (UID: \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 12:41:31.537228 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.536143 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk\" (UID: \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 12:41:31.537228 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.536218 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-metrics-certs\") pod \"maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk\" (UID: \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 12:41:31.537228 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.536339 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptj7s\" (UniqueName: \"kubernetes.io/projected/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-kube-api-access-ptj7s\") pod \"maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk\" (UID: \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 12:41:31.537228 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.536489 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk\" (UID: \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 12:41:31.537228 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:41:31.536695 4413 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-serving-cert: secret "maas-api-e2e-extmod-a-a786af-serving-cert" not found Oct 30 12:41:31.538364 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:41:31.538333 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-maas-api-tls podName:27ef9a37-c090-4cbb-824f-f0a9c54d3fe2 nodeName:}" failed. No retries permitted until 2026-10-30 12:41:32.038301312 +0000 UTC m=+2845.148144194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-maas-api-tls") pod "maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" (UID: "27ef9a37-c090-4cbb-824f-f0a9c54d3fe2") : secret "maas-api-e2e-extmod-a-a786af-serving-cert" not found Oct 30 12:41:31.539593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.539557 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk\" (UID: \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 12:41:31.541477 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.541429 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:31.543808 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.543784 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-metrics-certs\") pod \"maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk\" (UID: \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 12:41:31.566021 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.565985 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptj7s\" (UniqueName: \"kubernetes.io/projected/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-kube-api-access-ptj7s\") pod \"maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk\" (UID: \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 12:41:31.653218 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:31.653180 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:32.042710 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:32.042624 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk\" (UID: \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 12:41:32.045331 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:32.045299 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk\" (UID: \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 12:41:32.169634 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:32.169591 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:32.253130 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:32.253094 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 12:41:32.289752 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:32.289724 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:32.580328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:32.580299 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk"] Oct 30 12:41:32.580911 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:41:32.580887 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ef9a37_c090_4cbb_824f_f0a9c54d3fe2.slice/crio-24fec8daa20eab865c9d72f9d2db960b800267bf2821961e49a7610582bb181a WatchSource:0}: Error finding container 24fec8daa20eab865c9d72f9d2db960b800267bf2821961e49a7610582bb181a: Status 404 returned error can't find the container with id 24fec8daa20eab865c9d72f9d2db960b800267bf2821961e49a7610582bb181a Oct 30 12:41:32.872820 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:32.872734 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" event={"ID":"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2","Type":"ContainerStarted","Data":"24fec8daa20eab865c9d72f9d2db960b800267bf2821961e49a7610582bb181a"} Oct 30 12:41:33.310100 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:33.310059 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:33.882067 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:33.882028 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" event={"ID":"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2","Type":"ContainerStarted","Data":"09be7f1a5b4dcc460e7974aefdd4d5128c2edc9e9c51f5886cbd5424a74900c3"} Oct 30 12:41:33.882524 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:33.882231 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 12:41:33.926371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:33.926308 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" podStartSLOduration=2.536783469 podStartE2EDuration="2.926293117s" totalImagesPullingTime="389.509648ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:33.924182694 +0000 UTC m=+2847.034025576" watchObservedRunningTime="2026-10-30 12:41:33.926293117 +0000 UTC m=+2847.036135999" Oct 30 12:41:34.231364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:34.231275 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:34.357271 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:34.357224 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:34.503645 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:34.503561 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:35.266869 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:35.266830 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:35.817147 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:41:35.817109 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:41:36.974159 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:36.974128 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:38.418051 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.418004 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm"] Oct 30 12:41:38.422577 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.422550 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" Oct 30 12:41:38.448552 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.448521 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-6ad1da-serving-cert\"" Oct 30 12:41:38.453388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.453354 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm"] Oct 30 12:41:38.479300 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.479237 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/747e931b-c949-4428-afe2-16da8de97a9a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm\" (UID: \"747e931b-c949-4428-afe2-16da8de97a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" Oct 30 12:41:38.479487 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.479422 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747e931b-c949-4428-afe2-16da8de97a9a-metrics-certs\") pod \"maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm\" (UID: \"747e931b-c949-4428-afe2-16da8de97a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" Oct 30 12:41:38.479556 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.479541 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmmc5\" (UniqueName: \"kubernetes.io/projected/747e931b-c949-4428-afe2-16da8de97a9a-kube-api-access-bmmc5\") pod \"maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm\" (UID: \"747e931b-c949-4428-afe2-16da8de97a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" Oct 30 12:41:38.479663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.479648 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/747e931b-c949-4428-afe2-16da8de97a9a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm\" (UID: \"747e931b-c949-4428-afe2-16da8de97a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" Oct 30 12:41:38.517624 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.517587 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:38.581029 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.580994 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmmc5\" (UniqueName: \"kubernetes.io/projected/747e931b-c949-4428-afe2-16da8de97a9a-kube-api-access-bmmc5\") pod \"maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm\" (UID: \"747e931b-c949-4428-afe2-16da8de97a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" Oct 30 12:41:38.581243 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.581099 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/747e931b-c949-4428-afe2-16da8de97a9a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm\" (UID: \"747e931b-c949-4428-afe2-16da8de97a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" Oct 30 12:41:38.581243 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.581285 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/747e931b-c949-4428-afe2-16da8de97a9a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm\" (UID: \"747e931b-c949-4428-afe2-16da8de97a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" Oct 30 12:41:38.581441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.581365 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747e931b-c949-4428-afe2-16da8de97a9a-metrics-certs\") pod \"maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm\" (UID: \"747e931b-c949-4428-afe2-16da8de97a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" Oct 30 12:41:38.581867 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.581842 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/747e931b-c949-4428-afe2-16da8de97a9a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm\" (UID: \"747e931b-c949-4428-afe2-16da8de97a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" Oct 30 12:41:38.584301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.584243 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747e931b-c949-4428-afe2-16da8de97a9a-metrics-certs\") pod \"maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm\" (UID: \"747e931b-c949-4428-afe2-16da8de97a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" Oct 30 12:41:38.586750 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.586726 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/747e931b-c949-4428-afe2-16da8de97a9a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm\" (UID: \"747e931b-c949-4428-afe2-16da8de97a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" Oct 30 12:41:38.605419 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.605384 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmmc5\" (UniqueName: \"kubernetes.io/projected/747e931b-c949-4428-afe2-16da8de97a9a-kube-api-access-bmmc5\") pod \"maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm\" (UID: \"747e931b-c949-4428-afe2-16da8de97a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" Oct 30 12:41:38.639671 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.639619 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:38.737519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:38.737422 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" Oct 30 12:41:39.057725 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:39.053789 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm"] Oct 30 12:41:39.894292 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:39.894225 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 12:41:39.925019 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:39.924989 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" event={"ID":"747e931b-c949-4428-afe2-16da8de97a9a","Type":"ContainerStarted","Data":"6767f73c34057a9c1dadfd1073c88a4720fda9d1cdc8ce599fee507745703c09"} Oct 30 12:41:39.925196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:39.925045 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" event={"ID":"747e931b-c949-4428-afe2-16da8de97a9a","Type":"ContainerStarted","Data":"79af6a9063a0321877f26d926be1cb120503fcf03703a305c824658f903e766c"} Oct 30 12:41:39.925196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:39.925076 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" Oct 30 12:41:39.980119 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:39.980055 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" podStartSLOduration=1.648291792 podStartE2EDuration="1.980035131s" totalImagesPullingTime="331.743339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:39.978307818 +0000 UTC m=+2853.088150712" watchObservedRunningTime="2026-10-30 12:41:39.980035131 +0000 UTC m=+2853.089878013" Oct 30 12:41:40.308818 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.308695 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:40.343751 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.343715 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc"] Oct 30 12:41:40.348671 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.348649 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" Oct 30 12:41:40.368332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.365691 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-2545c7-serving-cert\"" Oct 30 12:41:40.371471 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.371334 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc"] Oct 30 12:41:40.441814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.441775 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:40.455274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.446431 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37eb395b-d510-4d1c-a59c-50d5d42b0efd-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc\" (UID: \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" Oct 30 12:41:40.455274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.446478 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37eb395b-d510-4d1c-a59c-50d5d42b0efd-metrics-certs\") pod \"maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc\" (UID: \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" Oct 30 12:41:40.455274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.446516 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37eb395b-d510-4d1c-a59c-50d5d42b0efd-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc\" (UID: \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" Oct 30 12:41:40.455274 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.446582 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v8qm\" (UniqueName: \"kubernetes.io/projected/37eb395b-d510-4d1c-a59c-50d5d42b0efd-kube-api-access-8v8qm\") pod \"maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc\" (UID: \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" Oct 30 12:41:40.555808 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.548670 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37eb395b-d510-4d1c-a59c-50d5d42b0efd-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc\" (UID: \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" Oct 30 12:41:40.555808 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.548754 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37eb395b-d510-4d1c-a59c-50d5d42b0efd-metrics-certs\") pod \"maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc\" (UID: \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" Oct 30 12:41:40.555808 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.548828 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37eb395b-d510-4d1c-a59c-50d5d42b0efd-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc\" (UID: \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" Oct 30 12:41:40.555808 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.548958 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v8qm\" (UniqueName: \"kubernetes.io/projected/37eb395b-d510-4d1c-a59c-50d5d42b0efd-kube-api-access-8v8qm\") pod \"maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc\" (UID: \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" Oct 30 12:41:40.560848 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.560772 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37eb395b-d510-4d1c-a59c-50d5d42b0efd-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc\" (UID: \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" Oct 30 12:41:40.577918 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.572936 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37eb395b-d510-4d1c-a59c-50d5d42b0efd-metrics-certs\") pod \"maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc\" (UID: \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" Oct 30 12:41:40.580778 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.580748 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:40.599634 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.598802 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37eb395b-d510-4d1c-a59c-50d5d42b0efd-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc\" (UID: \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" Oct 30 12:41:40.613789 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.613762 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v8qm\" (UniqueName: \"kubernetes.io/projected/37eb395b-d510-4d1c-a59c-50d5d42b0efd-kube-api-access-8v8qm\") pod \"maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc\" (UID: \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" Oct 30 12:41:40.666503 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.666459 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" Oct 30 12:41:40.713908 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.705826 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:40.993757 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:40.993731 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc"] Oct 30 12:41:40.995368 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:41:40.995341 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37eb395b_d510_4d1c_a59c_50d5d42b0efd.slice/crio-93d78b0f3befab5a89b16e8402a3152ffe2b9438f40b1d0943d90c17f571f567 WatchSource:0}: Error finding container 93d78b0f3befab5a89b16e8402a3152ffe2b9438f40b1d0943d90c17f571f567: Status 404 returned error can't find the container with id 93d78b0f3befab5a89b16e8402a3152ffe2b9438f40b1d0943d90c17f571f567 Oct 30 12:41:41.078421 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.078360 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:41.211469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.211429 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" podUID="72a8d280-04de-4a71-8036-dd8775bd9430" containerName="main" containerID="cri-o://e894acdf16a4f309a7443d61b101719f27356acf5c907a8d518e48fff1d2b917" gracePeriod=45 Oct 30 12:41:41.493994 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.493966 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:41:41.664182 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.664143 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:41.673569 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.673539 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a8d280-04de-4a71-8036-dd8775bd9430-kube-api-access-zfswt\" (UniqueName: \"kubernetes.io/projected/72a8d280-04de-4a71-8036-dd8775bd9430-kube-api-access-zfswt\") pod \"72a8d280-04de-4a71-8036-dd8775bd9430\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " Oct 30 12:41:41.673775 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.673585 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72a8d280-04de-4a71-8036-dd8775bd9430-tls-certs\" (UniqueName: \"kubernetes.io/secret/72a8d280-04de-4a71-8036-dd8775bd9430-tls-certs\") pod \"72a8d280-04de-4a71-8036-dd8775bd9430\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " Oct 30 12:41:41.673775 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.673629 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-dshm\") pod \"72a8d280-04de-4a71-8036-dd8775bd9430\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " Oct 30 12:41:41.673907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.673763 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-tmp-dir\") pod \"72a8d280-04de-4a71-8036-dd8775bd9430\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " Oct 30 12:41:41.674104 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.674073 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-tmp-dir" pod "72a8d280-04de-4a71-8036-dd8775bd9430" (UID: "72a8d280-04de-4a71-8036-dd8775bd9430"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:41.674532 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.674506 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-home" pod "72a8d280-04de-4a71-8036-dd8775bd9430" (UID: "72a8d280-04de-4a71-8036-dd8775bd9430"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:41.674638 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.674240 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-home\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-home\") pod \"72a8d280-04de-4a71-8036-dd8775bd9430\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " Oct 30 12:41:41.674708 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.674652 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-model-cache\") pod \"72a8d280-04de-4a71-8036-dd8775bd9430\" (UID: \"72a8d280-04de-4a71-8036-dd8775bd9430\") " Oct 30 12:41:41.674929 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.674905 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-model-cache" pod "72a8d280-04de-4a71-8036-dd8775bd9430" (UID: "72a8d280-04de-4a71-8036-dd8775bd9430"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:41.675815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.675791 4413 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:41.675917 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.675823 4413 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:41.675917 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.675838 4413 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:41.676315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.676295 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a8d280-04de-4a71-8036-dd8775bd9430-tls-certs" pod "72a8d280-04de-4a71-8036-dd8775bd9430" (UID: "72a8d280-04de-4a71-8036-dd8775bd9430"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:41:41.676532 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.676504 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-dshm" pod "72a8d280-04de-4a71-8036-dd8775bd9430" (UID: "72a8d280-04de-4a71-8036-dd8775bd9430"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:41:41.676642 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.676539 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a8d280-04de-4a71-8036-dd8775bd9430-kube-api-access-zfswt" pod "72a8d280-04de-4a71-8036-dd8775bd9430" (UID: "72a8d280-04de-4a71-8036-dd8775bd9430"). InnerVolumeSpecName "kube-api-access-zfswt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:41:41.781283 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.779789 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfswt\" (UniqueName: \"kubernetes.io/projected/72a8d280-04de-4a71-8036-dd8775bd9430-kube-api-access-zfswt\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:41.781283 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.779828 4413 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72a8d280-04de-4a71-8036-dd8775bd9430-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:41.781283 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.779845 4413 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a8d280-04de-4a71-8036-dd8775bd9430-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:41:41.940782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.940748 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" event={"ID":"37eb395b-d510-4d1c-a59c-50d5d42b0efd","Type":"ContainerStarted","Data":"765523985b3c8a0284ed58859f02613a305d56bf51fcdaf76e40b6baee5e17c5"} Oct 30 12:41:41.940981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.940803 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" event={"ID":"37eb395b-d510-4d1c-a59c-50d5d42b0efd","Type":"ContainerStarted","Data":"93d78b0f3befab5a89b16e8402a3152ffe2b9438f40b1d0943d90c17f571f567"} Oct 30 12:41:41.940981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.940838 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" Oct 30 12:41:41.942231 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.942205 4413 generic.go:356] "Generic (PLEG): container finished" podID="72a8d280-04de-4a71-8036-dd8775bd9430" containerID="e894acdf16a4f309a7443d61b101719f27356acf5c907a8d518e48fff1d2b917" exitCode=0 Oct 30 12:41:41.942401 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.942243 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" event={"ID":"72a8d280-04de-4a71-8036-dd8775bd9430","Type":"ContainerDied","Data":"e894acdf16a4f309a7443d61b101719f27356acf5c907a8d518e48fff1d2b917"} Oct 30 12:41:41.942401 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.942242 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" Oct 30 12:41:41.942401 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.942305 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf" event={"ID":"72a8d280-04de-4a71-8036-dd8775bd9430","Type":"ContainerDied","Data":"ed5a94a7e3aae8f5187c0a0f9ed4124aa99ce435d3481aaa2d4bda54f59bf05c"} Oct 30 12:41:41.942401 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.942327 4413 scope.go:122] "RemoveContainer" containerID="e894acdf16a4f309a7443d61b101719f27356acf5c907a8d518e48fff1d2b917" Oct 30 12:41:41.955073 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.955055 4413 scope.go:122] "RemoveContainer" containerID="e894acdf16a4f309a7443d61b101719f27356acf5c907a8d518e48fff1d2b917" Oct 30 12:41:41.955386 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:41:41.955364 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e894acdf16a4f309a7443d61b101719f27356acf5c907a8d518e48fff1d2b917\": container with ID starting with e894acdf16a4f309a7443d61b101719f27356acf5c907a8d518e48fff1d2b917 not found: ID does not exist" containerID="e894acdf16a4f309a7443d61b101719f27356acf5c907a8d518e48fff1d2b917" Oct 30 12:41:41.955468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.955397 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e894acdf16a4f309a7443d61b101719f27356acf5c907a8d518e48fff1d2b917"} err="failed to get container status \"e894acdf16a4f309a7443d61b101719f27356acf5c907a8d518e48fff1d2b917\": rpc error: code = NotFound desc = could not find container \"e894acdf16a4f309a7443d61b101719f27356acf5c907a8d518e48fff1d2b917\": container with ID starting with e894acdf16a4f309a7443d61b101719f27356acf5c907a8d518e48fff1d2b917 not found: ID does not exist" Oct 30 12:41:41.983573 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.983530 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" podStartSLOduration=1.625199386 podStartE2EDuration="1.983517382s" totalImagesPullingTime="358.317996ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:41.982430243 +0000 UTC m=+2855.092273124" watchObservedRunningTime="2026-10-30 12:41:41.983517382 +0000 UTC m=+2855.093360263" Oct 30 12:41:41.992820 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:41.992790 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:42.022045 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:42.022019 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf"] Oct 30 12:41:42.028259 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:42.028227 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-dbe123/auth-test-model-dbe123-kserve-865d75f9c-757gf"] Oct 30 12:41:42.397836 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:42.397781 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:43.020983 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:43.020945 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a8d280-04de-4a71-8036-dd8775bd9430" path="/var/lib/kubelet/pods/72a8d280-04de-4a71-8036-dd8775bd9430/volumes" Oct 30 12:41:44.113853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:44.113788 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:45.106264 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:45.106218 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:45.223652 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:45.223615 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:45.938365 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:45.938335 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" Oct 30 12:41:45.950020 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:45.949991 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:46.856300 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:46.856241 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:46.985467 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:46.985437 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:47.026697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:47.026666 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm"] Oct 30 12:41:47.027379 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:47.027363 4413 cpu_manager.go:414] "containerMap: removing container" podUID="72a8d280-04de-4a71-8036-dd8775bd9430" containerName="main" Oct 30 12:41:47.027437 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:47.027379 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a8d280-04de-4a71-8036-dd8775bd9430" containerName="main" Oct 30 12:41:47.027538 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:47.027530 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a8d280-04de-4a71-8036-dd8775bd9430" containerName="main" Oct 30 12:41:47.031228 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:47.031213 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm" Oct 30 12:41:47.056769 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:47.056564 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0816edc6-cc13-4ffb-87f8-8661aedfb0fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm\" (UID: \"0816edc6-cc13-4ffb-87f8-8661aedfb0fe\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm" Oct 30 12:41:47.057493 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:47.057459 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4279\" (UniqueName: \"kubernetes.io/projected/0816edc6-cc13-4ffb-87f8-8661aedfb0fe-kube-api-access-z4279\") pod \"payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm\" (UID: \"0816edc6-cc13-4ffb-87f8-8661aedfb0fe\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm" Oct 30 12:41:47.070212 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:47.070181 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm"] Oct 30 12:41:47.072591 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:47.072564 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-ac674b\"" Oct 30 12:41:47.072856 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:47.072833 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-ac674b-dockercfg-tm4xx\"" Oct 30 12:41:47.161442 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:47.161393 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0816edc6-cc13-4ffb-87f8-8661aedfb0fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm\" (UID: \"0816edc6-cc13-4ffb-87f8-8661aedfb0fe\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm" Oct 30 12:41:47.161923 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:47.161900 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4279\" (UniqueName: \"kubernetes.io/projected/0816edc6-cc13-4ffb-87f8-8661aedfb0fe-kube-api-access-z4279\") pod \"payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm\" (UID: \"0816edc6-cc13-4ffb-87f8-8661aedfb0fe\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm" Oct 30 12:41:47.163607 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:47.163582 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0816edc6-cc13-4ffb-87f8-8661aedfb0fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm\" (UID: \"0816edc6-cc13-4ffb-87f8-8661aedfb0fe\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm" Oct 30 12:41:47.238685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:47.238654 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4279\" (UniqueName: \"kubernetes.io/projected/0816edc6-cc13-4ffb-87f8-8661aedfb0fe-kube-api-access-z4279\") pod \"payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm\" (UID: \"0816edc6-cc13-4ffb-87f8-8661aedfb0fe\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm" Oct 30 12:41:47.345126 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:47.345087 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm" Oct 30 12:41:47.650924 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:47.650898 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm"] Oct 30 12:41:47.653208 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:41:47.653148 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0816edc6_cc13_4ffb_87f8_8661aedfb0fe.slice/crio-1b27742e0b0e567bf24c50a7c0e8c09d2933d38b24e6314f7c01e63cdbef79f2 WatchSource:0}: Error finding container 1b27742e0b0e567bf24c50a7c0e8c09d2933d38b24e6314f7c01e63cdbef79f2: Status 404 returned error can't find the container with id 1b27742e0b0e567bf24c50a7c0e8c09d2933d38b24e6314f7c01e63cdbef79f2 Oct 30 12:41:47.967624 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:47.967545 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" Oct 30 12:41:48.000830 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:48.000795 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm" event={"ID":"0816edc6-cc13-4ffb-87f8-8661aedfb0fe","Type":"ContainerStarted","Data":"989327640a7fc29aa47ea390815f9a931b57fc963cca4d80a8d6f4b3c520cde7"} Oct 30 12:41:48.001026 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:48.000870 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm" Oct 30 12:41:48.001026 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:48.000895 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm" event={"ID":"0816edc6-cc13-4ffb-87f8-8661aedfb0fe","Type":"ContainerStarted","Data":"1b27742e0b0e567bf24c50a7c0e8c09d2933d38b24e6314f7c01e63cdbef79f2"} Oct 30 12:41:48.061505 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:48.061453 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm" podStartSLOduration=1.061434979 podStartE2EDuration="1.061434979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:48.057392179 +0000 UTC m=+2861.167235062" watchObservedRunningTime="2026-10-30 12:41:48.061434979 +0000 UTC m=+2861.171277860" Oct 30 12:41:48.065493 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:48.065343 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:49.777636 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:49.777594 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:51.434461 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:51.434430 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:53.134982 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:53.134950 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:54.771174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:54.771140 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:56.451443 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:56.451408 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:58.037144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:58.037106 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:59.011182 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:59.011155 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm" Oct 30 12:41:59.552719 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:59.552675 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:41:59.702041 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:41:59.702014 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:01.340560 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:01.340521 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:02.950015 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:02.949981 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:04.495012 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:04.494980 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:04.623568 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:04.623540 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:04.739841 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:04.739809 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:05.816891 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:42:05.816854 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:42:06.416163 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:06.416129 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:08.365608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:08.365576 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:10.229727 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:10.229693 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:10.527592 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:10.527516 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:11.868318 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:11.868287 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:11.995099 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:11.995068 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:12.384584 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:12.384533 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:12.510138 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:12.510106 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:13.738663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:13.738623 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:15.572610 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:15.572582 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:16.706580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:16.706541 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:16.825178 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:16.825141 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:17.525348 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:17.525026 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:19.012894 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.012860 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:19.127450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.127414 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:19.151350 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.151316 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n"] Oct 30 12:42:19.157444 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.157421 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" Oct 30 12:42:19.182199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.182171 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n"] Oct 30 12:42:19.189664 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.189634 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-ac674b-serving-cert\"" Oct 30 12:42:19.219151 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.219094 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-maas-api-tls\") pod \"maas-api-e2e-autores-ac674b-b69698c54-pk74n\" (UID: \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" Oct 30 12:42:19.219374 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.219224 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhpd5\" (UniqueName: \"kubernetes.io/projected/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-kube-api-access-vhpd5\") pod \"maas-api-e2e-autores-ac674b-b69698c54-pk74n\" (UID: \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" Oct 30 12:42:19.219525 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.219504 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-openshift-service-ca\") pod \"maas-api-e2e-autores-ac674b-b69698c54-pk74n\" (UID: \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" Oct 30 12:42:19.219570 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.219552 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-metrics-certs\") pod \"maas-api-e2e-autores-ac674b-b69698c54-pk74n\" (UID: \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" Oct 30 12:42:19.260909 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.260875 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:19.266199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.266127 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg"] Oct 30 12:42:19.270823 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.270802 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" Oct 30 12:42:19.291647 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.291622 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0f7d14-dockercfg-k6ftm\"" Oct 30 12:42:19.295745 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.295722 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg"] Oct 30 12:42:19.315519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.315446 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0f7d14\"" Oct 30 12:42:19.320725 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.320698 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-openshift-service-ca\") pod \"maas-api-e2e-autores-ac674b-b69698c54-pk74n\" (UID: \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" Oct 30 12:42:19.320914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.320748 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-metrics-certs\") pod \"maas-api-e2e-autores-ac674b-b69698c54-pk74n\" (UID: \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" Oct 30 12:42:19.320914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.320881 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-maas-api-tls\") pod \"maas-api-e2e-autores-ac674b-b69698c54-pk74n\" (UID: \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" Oct 30 12:42:19.321104 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.320951 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhpd5\" (UniqueName: \"kubernetes.io/projected/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-kube-api-access-vhpd5\") pod \"maas-api-e2e-autores-ac674b-b69698c54-pk74n\" (UID: \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" Oct 30 12:42:19.322105 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.322049 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-openshift-service-ca\") pod \"maas-api-e2e-autores-ac674b-b69698c54-pk74n\" (UID: \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" Oct 30 12:42:19.324508 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.324482 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-metrics-certs\") pod \"maas-api-e2e-autores-ac674b-b69698c54-pk74n\" (UID: \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" Oct 30 12:42:19.324986 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.324966 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-maas-api-tls\") pod \"maas-api-e2e-autores-ac674b-b69698c54-pk74n\" (UID: \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" Oct 30 12:42:19.344433 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.344409 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhpd5\" (UniqueName: \"kubernetes.io/projected/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-kube-api-access-vhpd5\") pod \"maas-api-e2e-autores-ac674b-b69698c54-pk74n\" (UID: \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" Oct 30 12:42:19.373672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.373637 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:19.447273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.439799 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b11889-7268-40a1-8387-109af1464e04-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg\" (UID: \"29b11889-7268-40a1-8387-109af1464e04\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" Oct 30 12:42:19.447273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.439875 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkvc8\" (UniqueName: \"kubernetes.io/projected/29b11889-7268-40a1-8387-109af1464e04-kube-api-access-pkvc8\") pod \"payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg\" (UID: \"29b11889-7268-40a1-8387-109af1464e04\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" Oct 30 12:42:19.472086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.472048 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" Oct 30 12:42:19.541308 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.541182 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b11889-7268-40a1-8387-109af1464e04-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg\" (UID: \"29b11889-7268-40a1-8387-109af1464e04\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" Oct 30 12:42:19.541308 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.541229 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkvc8\" (UniqueName: \"kubernetes.io/projected/29b11889-7268-40a1-8387-109af1464e04-kube-api-access-pkvc8\") pod \"payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg\" (UID: \"29b11889-7268-40a1-8387-109af1464e04\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" Oct 30 12:42:19.542005 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.541983 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b11889-7268-40a1-8387-109af1464e04-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg\" (UID: \"29b11889-7268-40a1-8387-109af1464e04\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" Oct 30 12:42:19.574800 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.572121 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkvc8\" (UniqueName: \"kubernetes.io/projected/29b11889-7268-40a1-8387-109af1464e04-kube-api-access-pkvc8\") pod \"payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg\" (UID: \"29b11889-7268-40a1-8387-109af1464e04\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" Oct 30 12:42:19.585878 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.585852 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" Oct 30 12:42:19.818657 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.818626 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n"] Oct 30 12:42:19.819974 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:42:19.819952 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7b3c5e_ca33_453d_b0a9_9adc5ea8840c.slice/crio-2247917f1ede61f11b6201ce7db0d26461a3c90ccd17a4e586ea48317e37369f WatchSource:0}: Error finding container 2247917f1ede61f11b6201ce7db0d26461a3c90ccd17a4e586ea48317e37369f: Status 404 returned error can't find the container with id 2247917f1ede61f11b6201ce7db0d26461a3c90ccd17a4e586ea48317e37369f Oct 30 12:42:19.901130 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:19.901106 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg"] Oct 30 12:42:19.902726 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:42:19.902693 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b11889_7268_40a1_8387_109af1464e04.slice/crio-25f001ebe36d067c6a712645341747d6a994e92be02aeaf5f2136239763a66b1 WatchSource:0}: Error finding container 25f001ebe36d067c6a712645341747d6a994e92be02aeaf5f2136239763a66b1: Status 404 returned error can't find the container with id 25f001ebe36d067c6a712645341747d6a994e92be02aeaf5f2136239763a66b1 Oct 30 12:42:20.309107 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:20.309069 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" event={"ID":"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c","Type":"ContainerStarted","Data":"0c8002f4c633bfdb89b2bc382df9ca5d8fa232e15c46fed00c05029d3659d45f"} Oct 30 12:42:20.309747 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:20.309710 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" event={"ID":"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c","Type":"ContainerStarted","Data":"2247917f1ede61f11b6201ce7db0d26461a3c90ccd17a4e586ea48317e37369f"} Oct 30 12:42:20.310924 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:20.310902 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" Oct 30 12:42:20.316286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:20.315765 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" event={"ID":"29b11889-7268-40a1-8387-109af1464e04","Type":"ContainerStarted","Data":"e4453e893052b35169aedbd5791044a806d3a6a955ab98e47474d68f7bbb3f64"} Oct 30 12:42:20.316286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:20.315839 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" event={"ID":"29b11889-7268-40a1-8387-109af1464e04","Type":"ContainerStarted","Data":"25f001ebe36d067c6a712645341747d6a994e92be02aeaf5f2136239763a66b1"} Oct 30 12:42:20.316286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:20.315929 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" Oct 30 12:42:20.354635 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:20.354533 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" podStartSLOduration=1.087789014 podStartE2EDuration="1.354518194s" totalImagesPullingTime="266.72918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:20.3528148 +0000 UTC m=+2893.462657716" watchObservedRunningTime="2026-10-30 12:42:20.354518194 +0000 UTC m=+2893.464361075" Oct 30 12:42:20.394061 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:20.394007 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" podStartSLOduration=1.393992597 podStartE2EDuration="1.393992597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:20.39277077 +0000 UTC m=+2893.502613670" watchObservedRunningTime="2026-10-30 12:42:20.393992597 +0000 UTC m=+2893.503835477" Oct 30 12:42:20.553527 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:20.553496 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:21.174629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:21.174599 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:21.289902 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:21.289855 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:21.988559 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:21.988526 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:22.490889 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:22.490854 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:22.636277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:22.623769 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:22.827097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:22.827020 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:22.943568 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:22.943536 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:23.345969 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:23.345937 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:23.485060 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:23.485031 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:24.701447 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:24.701416 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:26.422406 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:26.422373 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:27.337910 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:27.337883 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" Oct 30 12:42:27.806510 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:27.806479 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:27.962976 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:27.962934 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:28.240405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:28.240368 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:28.373561 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:28.373530 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:28.492902 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:28.492825 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:29.689017 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:29.688980 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:29.829011 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:29.828969 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:29.892333 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:29.892303 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb"] Oct 30 12:42:29.898243 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:29.898213 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" Oct 30 12:42:29.926737 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:29.926708 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-12fd33-dockercfg-w8l7c\"" Oct 30 12:42:29.955487 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:29.932518 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-12fd33\"" Oct 30 12:42:30.022528 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:30.022489 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb"] Oct 30 12:42:30.092172 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:30.092140 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c9k6\" (UniqueName: \"kubernetes.io/projected/835438be-d2d1-40da-a7ed-cd420db69319-kube-api-access-2c9k6\") pod \"payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb\" (UID: \"835438be-d2d1-40da-a7ed-cd420db69319\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" Oct 30 12:42:30.092389 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:30.092297 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835438be-d2d1-40da-a7ed-cd420db69319-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb\" (UID: \"835438be-d2d1-40da-a7ed-cd420db69319\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" Oct 30 12:42:30.194361 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:30.194326 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9k6\" (UniqueName: \"kubernetes.io/projected/835438be-d2d1-40da-a7ed-cd420db69319-kube-api-access-2c9k6\") pod \"payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb\" (UID: \"835438be-d2d1-40da-a7ed-cd420db69319\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" Oct 30 12:42:30.194583 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:30.194452 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835438be-d2d1-40da-a7ed-cd420db69319-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb\" (UID: \"835438be-d2d1-40da-a7ed-cd420db69319\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" Oct 30 12:42:30.195194 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:30.195178 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835438be-d2d1-40da-a7ed-cd420db69319-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb\" (UID: \"835438be-d2d1-40da-a7ed-cd420db69319\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" Oct 30 12:42:30.217775 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:30.217705 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c9k6\" (UniqueName: \"kubernetes.io/projected/835438be-d2d1-40da-a7ed-cd420db69319-kube-api-access-2c9k6\") pod \"payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb\" (UID: \"835438be-d2d1-40da-a7ed-cd420db69319\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" Oct 30 12:42:30.448499 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:30.448442 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:30.514128 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:30.514037 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" Oct 30 12:42:30.891769 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:30.891730 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb"] Oct 30 12:42:30.892441 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:42:30.892409 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835438be_d2d1_40da_a7ed_cd420db69319.slice/crio-19a0faf29ccdefd783042b3c82a38290600ce699b6312790f27815a53e8822d4 WatchSource:0}: Error finding container 19a0faf29ccdefd783042b3c82a38290600ce699b6312790f27815a53e8822d4: Status 404 returned error can't find the container with id 19a0faf29ccdefd783042b3c82a38290600ce699b6312790f27815a53e8822d4 Oct 30 12:42:31.329369 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:31.329239 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" Oct 30 12:42:31.403735 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:31.403678 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" event={"ID":"835438be-d2d1-40da-a7ed-cd420db69319","Type":"ContainerStarted","Data":"f71d1c54fdd9603db20a4ee568489b69d82a344bd7de4b3051819e482d46d01c"} Oct 30 12:42:31.403978 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:31.403753 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" event={"ID":"835438be-d2d1-40da-a7ed-cd420db69319","Type":"ContainerStarted","Data":"19a0faf29ccdefd783042b3c82a38290600ce699b6312790f27815a53e8822d4"} Oct 30 12:42:31.403978 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:31.403786 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" Oct 30 12:42:31.444808 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:31.444752 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" podStartSLOduration=2.444731627 podStartE2EDuration="2.444731627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:31.442598106 +0000 UTC m=+2904.552440988" watchObservedRunningTime="2026-10-30 12:42:31.444731627 +0000 UTC m=+2904.554574508" Oct 30 12:42:32.040974 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:32.040938 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:32.151882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:32.151842 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:32.291843 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:32.291757 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:33.849982 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:33.849946 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:35.470237 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:35.470205 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:35.815491 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:42:35.815407 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:42:37.206360 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:37.206325 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:38.778031 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:38.777989 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:40.475747 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:40.475713 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:42.177363 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:42.177330 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:42.416283 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:42.416223 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" Oct 30 12:42:43.908543 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:43.908507 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:45.931765 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:45.931731 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:47.697160 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:47.697131 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:48.969472 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:48.969402 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:49.523935 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:49.523903 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:50.432666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:50.432634 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:51.150180 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:51.150151 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:51.296739 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:51.296709 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:51.573700 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:51.573597 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:53.297040 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:53.297000 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:53.425536 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:53.425495 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:53.554005 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:53.553913 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:54.975050 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:54.975016 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:55.118146 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:55.118084 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:55.309651 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:55.309574 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:56.820485 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:56.820457 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:56.966358 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:56.966325 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9"] Oct 30 12:42:56.973081 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:56.973059 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" Oct 30 12:42:56.990066 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:56.990045 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-ecb3aa-dockercfg-5mf57\"" Oct 30 12:42:56.993224 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:56.993211 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-ecb3aa\"" Oct 30 12:42:57.007585 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:57.007553 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9"] Oct 30 12:42:57.010164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:57.010138 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:57.086484 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:57.086386 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c59d84af-0bbf-4a53-bbe0-07e80541bb86-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9\" (UID: \"c59d84af-0bbf-4a53-bbe0-07e80541bb86\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" Oct 30 12:42:57.093273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:57.087822 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5b7c\" (UniqueName: \"kubernetes.io/projected/c59d84af-0bbf-4a53-bbe0-07e80541bb86-kube-api-access-x5b7c\") pod \"payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9\" (UID: \"c59d84af-0bbf-4a53-bbe0-07e80541bb86\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" Oct 30 12:42:57.214709 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:57.214670 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c59d84af-0bbf-4a53-bbe0-07e80541bb86-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9\" (UID: \"c59d84af-0bbf-4a53-bbe0-07e80541bb86\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" Oct 30 12:42:57.214918 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:57.214780 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5b7c\" (UniqueName: \"kubernetes.io/projected/c59d84af-0bbf-4a53-bbe0-07e80541bb86-kube-api-access-x5b7c\") pod \"payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9\" (UID: \"c59d84af-0bbf-4a53-bbe0-07e80541bb86\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" Oct 30 12:42:57.215593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:57.215568 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c59d84af-0bbf-4a53-bbe0-07e80541bb86-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9\" (UID: \"c59d84af-0bbf-4a53-bbe0-07e80541bb86\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" Oct 30 12:42:57.238068 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:57.238039 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5b7c\" (UniqueName: \"kubernetes.io/projected/c59d84af-0bbf-4a53-bbe0-07e80541bb86-kube-api-access-x5b7c\") pod \"payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9\" (UID: \"c59d84af-0bbf-4a53-bbe0-07e80541bb86\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" Oct 30 12:42:57.287502 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:57.287471 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" Oct 30 12:42:57.605118 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:57.605088 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9"] Oct 30 12:42:57.606494 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:42:57.606457 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59d84af_0bbf_4a53_bbe0_07e80541bb86.slice/crio-0dc104249ce5e5ff7d80be119e823aa7a6e210cc51a494ab81148b1f74b48c5b WatchSource:0}: Error finding container 0dc104249ce5e5ff7d80be119e823aa7a6e210cc51a494ab81148b1f74b48c5b: Status 404 returned error can't find the container with id 0dc104249ce5e5ff7d80be119e823aa7a6e210cc51a494ab81148b1f74b48c5b Oct 30 12:42:57.621558 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:57.621526 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" event={"ID":"c59d84af-0bbf-4a53-bbe0-07e80541bb86","Type":"ContainerStarted","Data":"0dc104249ce5e5ff7d80be119e823aa7a6e210cc51a494ab81148b1f74b48c5b"} Oct 30 12:42:57.679234 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:57.679192 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:57.949090 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:57.949050 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:58.076239 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:58.076198 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:58.631144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:58.631107 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" event={"ID":"c59d84af-0bbf-4a53-bbe0-07e80541bb86","Type":"ContainerStarted","Data":"cce967bc67ef68aa5ae6ffdf3dc40be89cc667c2b67a98f842248ef7a5e87128"} Oct 30 12:42:58.631400 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:58.631381 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" Oct 30 12:42:58.677896 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:58.677836 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" podStartSLOduration=2.677821677 podStartE2EDuration="2.677821677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:42:58.676505645 +0000 UTC m=+2931.786348527" watchObservedRunningTime="2026-10-30 12:42:58.677821677 +0000 UTC m=+2931.787664558" Oct 30 12:42:59.491424 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:59.491386 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:42:59.746727 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:42:59.746649 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:01.302067 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:01.302037 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:01.859362 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:01.859330 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:03.243745 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:03.243713 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:04.984432 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:04.984400 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:05.510125 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:05.510094 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:05.821307 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:43:05.821189 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:43:06.796954 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:06.796922 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:08.593663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:08.593623 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:08.750197 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:08.750167 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:09.642035 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:09.642002 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" Oct 30 12:43:10.328234 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.328202 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv"] Oct 30 12:43:10.332926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.332906 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" Oct 30 12:43:10.355835 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.355806 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-0de4b9-serving-cert\"" Oct 30 12:43:10.366023 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.365994 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv"] Oct 30 12:43:10.413495 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.413460 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdh6k\" (UniqueName: \"kubernetes.io/projected/4c30ec11-6046-4da2-944a-c45f40848016-kube-api-access-fdh6k\") pod \"maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv\" (UID: \"4c30ec11-6046-4da2-944a-c45f40848016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" Oct 30 12:43:10.413684 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.413528 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c30ec11-6046-4da2-944a-c45f40848016-metrics-certs\") pod \"maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv\" (UID: \"4c30ec11-6046-4da2-944a-c45f40848016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" Oct 30 12:43:10.413684 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.413630 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c30ec11-6046-4da2-944a-c45f40848016-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv\" (UID: \"4c30ec11-6046-4da2-944a-c45f40848016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" Oct 30 12:43:10.413782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.413759 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c30ec11-6046-4da2-944a-c45f40848016-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv\" (UID: \"4c30ec11-6046-4da2-944a-c45f40848016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" Oct 30 12:43:10.438485 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.438453 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:10.515556 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.515527 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c30ec11-6046-4da2-944a-c45f40848016-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv\" (UID: \"4c30ec11-6046-4da2-944a-c45f40848016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" Oct 30 12:43:10.515731 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.515642 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c30ec11-6046-4da2-944a-c45f40848016-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv\" (UID: \"4c30ec11-6046-4da2-944a-c45f40848016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" Oct 30 12:43:10.515892 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.515865 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdh6k\" (UniqueName: \"kubernetes.io/projected/4c30ec11-6046-4da2-944a-c45f40848016-kube-api-access-fdh6k\") pod \"maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv\" (UID: \"4c30ec11-6046-4da2-944a-c45f40848016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" Oct 30 12:43:10.516054 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.515935 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c30ec11-6046-4da2-944a-c45f40848016-metrics-certs\") pod \"maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv\" (UID: \"4c30ec11-6046-4da2-944a-c45f40848016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" Oct 30 12:43:10.516467 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.516447 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c30ec11-6046-4da2-944a-c45f40848016-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv\" (UID: \"4c30ec11-6046-4da2-944a-c45f40848016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" Oct 30 12:43:10.518363 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.518346 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c30ec11-6046-4da2-944a-c45f40848016-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv\" (UID: \"4c30ec11-6046-4da2-944a-c45f40848016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" Oct 30 12:43:10.518628 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.518608 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c30ec11-6046-4da2-944a-c45f40848016-metrics-certs\") pod \"maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv\" (UID: \"4c30ec11-6046-4da2-944a-c45f40848016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" Oct 30 12:43:10.540885 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.540856 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdh6k\" (UniqueName: \"kubernetes.io/projected/4c30ec11-6046-4da2-944a-c45f40848016-kube-api-access-fdh6k\") pod \"maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv\" (UID: \"4c30ec11-6046-4da2-944a-c45f40848016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" Oct 30 12:43:10.606773 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.606688 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:10.647865 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.647837 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" Oct 30 12:43:10.963077 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:10.963054 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv"] Oct 30 12:43:10.964686 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:43:10.964659 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c30ec11_6046_4da2_944a_c45f40848016.slice/crio-4e592f9dbb339bb7161dd7382e852e27ac8e0c161297278594a291cc04a8bba4 WatchSource:0}: Error finding container 4e592f9dbb339bb7161dd7382e852e27ac8e0c161297278594a291cc04a8bba4: Status 404 returned error can't find the container with id 4e592f9dbb339bb7161dd7382e852e27ac8e0c161297278594a291cc04a8bba4 Oct 30 12:43:11.761016 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:11.760917 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" event={"ID":"4c30ec11-6046-4da2-944a-c45f40848016","Type":"ContainerStarted","Data":"4ed9a101ac6a1e51189410919d185a34cf3ef3f1a226d06c7e39bc0ce7de39e2"} Oct 30 12:43:11.761016 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:11.760972 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" event={"ID":"4c30ec11-6046-4da2-944a-c45f40848016","Type":"ContainerStarted","Data":"4e592f9dbb339bb7161dd7382e852e27ac8e0c161297278594a291cc04a8bba4"} Oct 30 12:43:11.761461 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:11.761057 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" Oct 30 12:43:11.812647 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:11.812594 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" podStartSLOduration=1.386047088 podStartE2EDuration="1.812580264s" totalImagesPullingTime="426.533176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:43:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:43:11.808957814 +0000 UTC m=+2944.918800712" watchObservedRunningTime="2026-10-30 12:43:11.812580264 +0000 UTC m=+2944.922423144" Oct 30 12:43:12.300662 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:12.300630 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:14.098134 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:14.098086 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:15.887849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:15.887812 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:17.680650 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:17.680618 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:17.776436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:17.776407 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" Oct 30 12:43:19.540495 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:19.540459 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:21.212779 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:21.212730 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:22.849717 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:22.849678 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm"] Oct 30 12:43:22.850109 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:22.849972 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" podUID="e0ce1cc0-165b-433d-88bc-719f2f95c40d" containerName="istio-proxy" containerID="cri-o://e4cf1415150d4499116a7efaa2b874d4166ab4787e892ced60926ffa4bb7211e" gracePeriod=30 Oct 30 12:43:22.971646 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:22.971614 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:24.683282 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:24.683209 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:26.626932 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:26.626902 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:28.124871 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.124847 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:43:28.190001 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.189969 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-envoy\") pod \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " Oct 30 12:43:28.190182 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.190041 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-workload-certs\") pod \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " Oct 30 12:43:28.190182 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.190083 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-credential-socket\") pod \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " Oct 30 12:43:28.190182 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.190116 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-workload-socket\") pod \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " Oct 30 12:43:28.190381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.190324 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-podinfo\") pod \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " Oct 30 12:43:28.190444 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.190425 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istiod-ca-cert\") pod \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " Oct 30 12:43:28.190499 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.190487 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-token\" (UniqueName: \"kubernetes.io/projected/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-token\") pod \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " Oct 30 12:43:28.190571 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.190556 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-data\") pod \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " Oct 30 12:43:28.190631 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.190618 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ce1cc0-165b-433d-88bc-719f2f95c40d-kube-api-access-zd9hv\" (UniqueName: \"kubernetes.io/projected/e0ce1cc0-165b-433d-88bc-719f2f95c40d-kube-api-access-zd9hv\") pod \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\" (UID: \"e0ce1cc0-165b-433d-88bc-719f2f95c40d\") " Oct 30 12:43:28.190692 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.190623 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-credential-socket" pod "e0ce1cc0-165b-433d-88bc-719f2f95c40d" (UID: "e0ce1cc0-165b-433d-88bc-719f2f95c40d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:28.191218 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.190929 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istiod-ca-cert" pod "e0ce1cc0-165b-433d-88bc-719f2f95c40d" (UID: "e0ce1cc0-165b-433d-88bc-719f2f95c40d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:43:28.191405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.191222 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-workload-socket" pod "e0ce1cc0-165b-433d-88bc-719f2f95c40d" (UID: "e0ce1cc0-165b-433d-88bc-719f2f95c40d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:28.191488 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.191429 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-data" pod "e0ce1cc0-165b-433d-88bc-719f2f95c40d" (UID: "e0ce1cc0-165b-433d-88bc-719f2f95c40d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:28.192117 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.192028 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:43:28.192117 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.192057 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:43:28.192117 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.192074 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:43:28.192478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.192446 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-workload-certs" pod "e0ce1cc0-165b-433d-88bc-719f2f95c40d" (UID: "e0ce1cc0-165b-433d-88bc-719f2f95c40d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:28.192760 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.192713 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-envoy" pod "e0ce1cc0-165b-433d-88bc-719f2f95c40d" (UID: "e0ce1cc0-165b-433d-88bc-719f2f95c40d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:43:28.194644 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.194617 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ce1cc0-165b-433d-88bc-719f2f95c40d-kube-api-access-zd9hv" pod "e0ce1cc0-165b-433d-88bc-719f2f95c40d" (UID: "e0ce1cc0-165b-433d-88bc-719f2f95c40d"). InnerVolumeSpecName "kube-api-access-zd9hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:28.194959 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.194670 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-token" pod "e0ce1cc0-165b-433d-88bc-719f2f95c40d" (UID: "e0ce1cc0-165b-433d-88bc-719f2f95c40d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:28.195162 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.195138 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-podinfo" pod "e0ce1cc0-165b-433d-88bc-719f2f95c40d" (UID: "e0ce1cc0-165b-433d-88bc-719f2f95c40d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:43:28.294030 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.294003 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:43:28.294030 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.294029 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:43:28.294328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.294039 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd9hv\" (UniqueName: \"kubernetes.io/projected/e0ce1cc0-165b-433d-88bc-719f2f95c40d-kube-api-access-zd9hv\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:43:28.294328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.294047 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:43:28.294328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.294056 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:43:28.294328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.294064 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ce1cc0-165b-433d-88bc-719f2f95c40d-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:43:28.401982 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.401912 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:28.950702 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.950669 4413 generic.go:356] "Generic (PLEG): container finished" podID="e0ce1cc0-165b-433d-88bc-719f2f95c40d" containerID="e4cf1415150d4499116a7efaa2b874d4166ab4787e892ced60926ffa4bb7211e" exitCode=0 Oct 30 12:43:28.950869 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.950717 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" Oct 30 12:43:28.950869 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.950748 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" event={"ID":"e0ce1cc0-165b-433d-88bc-719f2f95c40d","Type":"ContainerDied","Data":"e4cf1415150d4499116a7efaa2b874d4166ab4787e892ced60926ffa4bb7211e"} Oct 30 12:43:28.950869 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.950797 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm" event={"ID":"e0ce1cc0-165b-433d-88bc-719f2f95c40d","Type":"ContainerDied","Data":"3a89f7d2f62854685e6ad482049382b7d6fbf46f9298d60c1e9d77078832de4e"} Oct 30 12:43:28.950869 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.950820 4413 scope.go:122] "RemoveContainer" containerID="e4cf1415150d4499116a7efaa2b874d4166ab4787e892ced60926ffa4bb7211e" Oct 30 12:43:28.965889 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.965871 4413 scope.go:122] "RemoveContainer" containerID="e4cf1415150d4499116a7efaa2b874d4166ab4787e892ced60926ffa4bb7211e" Oct 30 12:43:28.967468 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:43:28.967442 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4cf1415150d4499116a7efaa2b874d4166ab4787e892ced60926ffa4bb7211e\": container with ID starting with e4cf1415150d4499116a7efaa2b874d4166ab4787e892ced60926ffa4bb7211e not found: ID does not exist" containerID="e4cf1415150d4499116a7efaa2b874d4166ab4787e892ced60926ffa4bb7211e" Oct 30 12:43:28.967559 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:28.967481 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4cf1415150d4499116a7efaa2b874d4166ab4787e892ced60926ffa4bb7211e"} err="failed to get container status \"e4cf1415150d4499116a7efaa2b874d4166ab4787e892ced60926ffa4bb7211e\": rpc error: code = NotFound desc = could not find container \"e4cf1415150d4499116a7efaa2b874d4166ab4787e892ced60926ffa4bb7211e\": container with ID starting with e4cf1415150d4499116a7efaa2b874d4166ab4787e892ced60926ffa4bb7211e not found: ID does not exist" Oct 30 12:43:29.031992 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:29.031968 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm"] Oct 30 12:43:29.043678 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:29.043656 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-lblbm"] Oct 30 12:43:30.217144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:30.217098 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:31.020610 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:31.020577 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ce1cc0-165b-433d-88bc-719f2f95c40d" path="/var/lib/kubelet/pods/e0ce1cc0-165b-433d-88bc-719f2f95c40d/volumes" Oct 30 12:43:31.956276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:31.956222 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:33.700642 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:33.700603 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:35.744188 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:35.744151 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:35.818382 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:43:35.818332 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:43:35.902761 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:35.902701 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:37.708635 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:37.708596 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:39.451179 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.451146 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479"] Oct 30 12:43:39.451944 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.451926 4413 cpu_manager.go:414] "containerMap: removing container" podUID="e0ce1cc0-165b-433d-88bc-719f2f95c40d" containerName="istio-proxy" Oct 30 12:43:39.451989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.451945 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ce1cc0-165b-433d-88bc-719f2f95c40d" containerName="istio-proxy" Oct 30 12:43:39.452076 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.452067 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ce1cc0-165b-433d-88bc-719f2f95c40d" containerName="istio-proxy" Oct 30 12:43:39.455918 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.455900 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" Oct 30 12:43:39.475230 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.475207 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-ecb3aa-serving-cert\"" Oct 30 12:43:39.492354 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.492324 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479"] Oct 30 12:43:39.555571 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.555541 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c843f24-0220-4063-8d6d-d1f84196d9b2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479\" (UID: \"8c843f24-0220-4063-8d6d-d1f84196d9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" Oct 30 12:43:39.555736 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.555671 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c843f24-0220-4063-8d6d-d1f84196d9b2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479\" (UID: \"8c843f24-0220-4063-8d6d-d1f84196d9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" Oct 30 12:43:39.555782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.555733 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m7bm\" (UniqueName: \"kubernetes.io/projected/8c843f24-0220-4063-8d6d-d1f84196d9b2-kube-api-access-8m7bm\") pod \"maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479\" (UID: \"8c843f24-0220-4063-8d6d-d1f84196d9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" Oct 30 12:43:39.555853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.555837 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c843f24-0220-4063-8d6d-d1f84196d9b2-metrics-certs\") pod \"maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479\" (UID: \"8c843f24-0220-4063-8d6d-d1f84196d9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" Oct 30 12:43:39.557052 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.557031 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:39.658823 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.658784 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m7bm\" (UniqueName: \"kubernetes.io/projected/8c843f24-0220-4063-8d6d-d1f84196d9b2-kube-api-access-8m7bm\") pod \"maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479\" (UID: \"8c843f24-0220-4063-8d6d-d1f84196d9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" Oct 30 12:43:39.659051 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.658894 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c843f24-0220-4063-8d6d-d1f84196d9b2-metrics-certs\") pod \"maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479\" (UID: \"8c843f24-0220-4063-8d6d-d1f84196d9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" Oct 30 12:43:39.660221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.660167 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c843f24-0220-4063-8d6d-d1f84196d9b2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479\" (UID: \"8c843f24-0220-4063-8d6d-d1f84196d9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" Oct 30 12:43:39.660415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.660396 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c843f24-0220-4063-8d6d-d1f84196d9b2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479\" (UID: \"8c843f24-0220-4063-8d6d-d1f84196d9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" Oct 30 12:43:39.661365 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.661342 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c843f24-0220-4063-8d6d-d1f84196d9b2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479\" (UID: \"8c843f24-0220-4063-8d6d-d1f84196d9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" Oct 30 12:43:39.663054 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.663025 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c843f24-0220-4063-8d6d-d1f84196d9b2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479\" (UID: \"8c843f24-0220-4063-8d6d-d1f84196d9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" Oct 30 12:43:39.663821 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.663805 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c843f24-0220-4063-8d6d-d1f84196d9b2-metrics-certs\") pod \"maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479\" (UID: \"8c843f24-0220-4063-8d6d-d1f84196d9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" Oct 30 12:43:39.685513 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.685480 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m7bm\" (UniqueName: \"kubernetes.io/projected/8c843f24-0220-4063-8d6d-d1f84196d9b2-kube-api-access-8m7bm\") pod \"maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479\" (UID: \"8c843f24-0220-4063-8d6d-d1f84196d9b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" Oct 30 12:43:39.715343 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.715155 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:39.770407 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.770369 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" Oct 30 12:43:39.896674 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:39.896645 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:40.088381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:40.088356 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479"] Oct 30 12:43:40.089222 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:43:40.089196 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c843f24_0220_4063_8d6d_d1f84196d9b2.slice/crio-71b063a271178352c9a518d027d20b6580f6e795527a86afecafc47450434870 WatchSource:0}: Error finding container 71b063a271178352c9a518d027d20b6580f6e795527a86afecafc47450434870: Status 404 returned error can't find the container with id 71b063a271178352c9a518d027d20b6580f6e795527a86afecafc47450434870 Oct 30 12:43:41.061921 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:41.061887 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" event={"ID":"8c843f24-0220-4063-8d6d-d1f84196d9b2","Type":"ContainerStarted","Data":"a43fe32318610c421d56f6785362614143c4bce112bb7e64cc52422e85efecf4"} Oct 30 12:43:41.062345 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:41.061945 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" event={"ID":"8c843f24-0220-4063-8d6d-d1f84196d9b2","Type":"ContainerStarted","Data":"71b063a271178352c9a518d027d20b6580f6e795527a86afecafc47450434870"} Oct 30 12:43:41.062345 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:41.061974 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" Oct 30 12:43:41.107219 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:41.107167 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" podStartSLOduration=1.626633483 podStartE2EDuration="2.107152511s" totalImagesPullingTime="480.519028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:43:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:43:41.106522553 +0000 UTC m=+2974.216365435" watchObservedRunningTime="2026-10-30 12:43:41.107152511 +0000 UTC m=+2974.216995391" Oct 30 12:43:41.571757 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:41.571712 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:43.374554 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:43.374487 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:43.523017 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:43.522978 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:44.746195 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:44.746163 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:44.902227 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:44.902200 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:45.066405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:45.066328 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:45.216907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:45.216880 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:45.376372 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:45.376343 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:45.549711 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:45.549675 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:45.747660 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:45.747575 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:46.337832 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:46.337804 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:46.437804 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:46.437771 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm"] Oct 30 12:43:46.506728 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:46.506696 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:46.701002 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:46.700960 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:47.074692 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:47.074621 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" Oct 30 12:43:47.698100 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:47.698073 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:47.874125 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:47.874094 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:48.079851 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.079768 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2"] Oct 30 12:43:48.088641 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.088616 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" Oct 30 12:43:48.104078 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.104055 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2"] Oct 30 12:43:48.108404 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.108378 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0f7d14\"/\"agc-cred-260a8fead43c\"" Oct 30 12:43:48.111726 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.111705 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f7d14\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0f7d14\"" Oct 30 12:43:48.111955 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.111917 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f7d14\"/\"openshift-service-ca.crt\"" Oct 30 12:43:48.112188 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.112129 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f7d14\"/\"kube-root-ca.crt\"" Oct 30 12:43:48.112292 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.112203 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0f7d14\"/\"payload-processing-external-model-e2e--d59b9242-dockercfg-4bzwv\"" Oct 30 12:43:48.115297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.112420 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f7d14\"/\"routing-overlay\"" Oct 30 12:43:48.120544 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.120517 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2\" (UID: \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" Oct 30 12:43:48.120681 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.120658 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2\" (UID: \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" Oct 30 12:43:48.120762 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.120735 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2\" (UID: \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" Oct 30 12:43:48.120878 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.120861 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt4cq\" (UniqueName: \"kubernetes.io/projected/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-kube-api-access-tt4cq\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2\" (UID: \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" Oct 30 12:43:48.222446 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.222413 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2\" (UID: \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" Oct 30 12:43:48.222622 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.222466 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2\" (UID: \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" Oct 30 12:43:48.222622 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.222510 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2\" (UID: \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" Oct 30 12:43:48.222622 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.222572 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt4cq\" (UniqueName: \"kubernetes.io/projected/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-kube-api-access-tt4cq\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2\" (UID: \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" Oct 30 12:43:48.223231 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.223209 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2\" (UID: \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" Oct 30 12:43:48.223319 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.223301 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2\" (UID: \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" Oct 30 12:43:48.225043 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.225024 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2\" (UID: \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" Oct 30 12:43:48.245939 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.245905 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt4cq\" (UniqueName: \"kubernetes.io/projected/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-kube-api-access-tt4cq\") pod \"payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2\" (UID: \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\") " pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" Oct 30 12:43:48.405469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.405433 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" Oct 30 12:43:48.725468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:48.725438 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2"] Oct 30 12:43:48.726104 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:43:48.726076 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad807e4_0d66_4856_964a_5b7a4a3a80bb.slice/crio-aa538f650a79d0f63d5f03eb12219f9fcaa128280b08ef6f9ae865173b2bf140 WatchSource:0}: Error finding container aa538f650a79d0f63d5f03eb12219f9fcaa128280b08ef6f9ae865173b2bf140: Status 404 returned error can't find the container with id aa538f650a79d0f63d5f03eb12219f9fcaa128280b08ef6f9ae865173b2bf140 Oct 30 12:43:49.126557 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:49.126524 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" event={"ID":"5ad807e4-0d66-4856-964a-5b7a4a3a80bb","Type":"ContainerStarted","Data":"1e9ded4470f50828b2d0be864d90d08c1b8432d986256111a7f944b0082fb401"} Oct 30 12:43:49.126956 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:49.126585 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" event={"ID":"5ad807e4-0d66-4856-964a-5b7a4a3a80bb","Type":"ContainerStarted","Data":"aa538f650a79d0f63d5f03eb12219f9fcaa128280b08ef6f9ae865173b2bf140"} Oct 30 12:43:49.126956 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:49.126724 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" Oct 30 12:43:49.169732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:49.169680 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" podStartSLOduration=1.169667677 podStartE2EDuration="1.169667677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:43:49.167339502 +0000 UTC m=+2982.277182384" watchObservedRunningTime="2026-10-30 12:43:49.169667677 +0000 UTC m=+2982.279510559" Oct 30 12:43:49.455815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:49.455705 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:51.226936 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:51.226900 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:51.438821 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:51.438779 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm" podUID="0816edc6-cc13-4ffb-87f8-8661aedfb0fe" containerName="payload-pre-processing" containerID="cri-o://989327640a7fc29aa47ea390815f9a931b57fc963cca4d80a8d6f4b3c520cde7" gracePeriod=25 Oct 30 12:43:51.719020 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:51.718984 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm" Oct 30 12:43:51.789268 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:51.789222 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0816edc6-cc13-4ffb-87f8-8661aedfb0fe-kube-api-access-z4279\" (UniqueName: \"kubernetes.io/projected/0816edc6-cc13-4ffb-87f8-8661aedfb0fe-kube-api-access-z4279\") pod \"0816edc6-cc13-4ffb-87f8-8661aedfb0fe\" (UID: \"0816edc6-cc13-4ffb-87f8-8661aedfb0fe\") " Oct 30 12:43:51.789458 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:51.789423 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0816edc6-cc13-4ffb-87f8-8661aedfb0fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0816edc6-cc13-4ffb-87f8-8661aedfb0fe-plugins-config-volume\") pod \"0816edc6-cc13-4ffb-87f8-8661aedfb0fe\" (UID: \"0816edc6-cc13-4ffb-87f8-8661aedfb0fe\") " Oct 30 12:43:51.789765 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:51.789743 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0816edc6-cc13-4ffb-87f8-8661aedfb0fe-plugins-config-volume" pod "0816edc6-cc13-4ffb-87f8-8661aedfb0fe" (UID: "0816edc6-cc13-4ffb-87f8-8661aedfb0fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:43:51.790051 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:51.790031 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0816edc6-cc13-4ffb-87f8-8661aedfb0fe-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:43:51.791584 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:51.791565 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0816edc6-cc13-4ffb-87f8-8661aedfb0fe-kube-api-access-z4279" pod "0816edc6-cc13-4ffb-87f8-8661aedfb0fe" (UID: "0816edc6-cc13-4ffb-87f8-8661aedfb0fe"). InnerVolumeSpecName "kube-api-access-z4279". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:43:51.891632 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:51.891606 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4279\" (UniqueName: \"kubernetes.io/projected/0816edc6-cc13-4ffb-87f8-8661aedfb0fe-kube-api-access-z4279\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:43:52.149643 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:52.149600 4413 generic.go:356] "Generic (PLEG): container finished" podID="0816edc6-cc13-4ffb-87f8-8661aedfb0fe" containerID="989327640a7fc29aa47ea390815f9a931b57fc963cca4d80a8d6f4b3c520cde7" exitCode=0 Oct 30 12:43:52.149643 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:52.149648 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm" Oct 30 12:43:52.149876 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:52.149682 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm" event={"ID":"0816edc6-cc13-4ffb-87f8-8661aedfb0fe","Type":"ContainerDied","Data":"989327640a7fc29aa47ea390815f9a931b57fc963cca4d80a8d6f4b3c520cde7"} Oct 30 12:43:52.149876 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:52.149729 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm" event={"ID":"0816edc6-cc13-4ffb-87f8-8661aedfb0fe","Type":"ContainerDied","Data":"1b27742e0b0e567bf24c50a7c0e8c09d2933d38b24e6314f7c01e63cdbef79f2"} Oct 30 12:43:52.149876 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:52.149749 4413 scope.go:122] "RemoveContainer" containerID="989327640a7fc29aa47ea390815f9a931b57fc963cca4d80a8d6f4b3c520cde7" Oct 30 12:43:52.163552 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:52.163533 4413 scope.go:122] "RemoveContainer" containerID="989327640a7fc29aa47ea390815f9a931b57fc963cca4d80a8d6f4b3c520cde7" Oct 30 12:43:52.163794 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:43:52.163777 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989327640a7fc29aa47ea390815f9a931b57fc963cca4d80a8d6f4b3c520cde7\": container with ID starting with 989327640a7fc29aa47ea390815f9a931b57fc963cca4d80a8d6f4b3c520cde7 not found: ID does not exist" containerID="989327640a7fc29aa47ea390815f9a931b57fc963cca4d80a8d6f4b3c520cde7" Oct 30 12:43:52.163876 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:52.163804 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989327640a7fc29aa47ea390815f9a931b57fc963cca4d80a8d6f4b3c520cde7"} err="failed to get container status \"989327640a7fc29aa47ea390815f9a931b57fc963cca4d80a8d6f4b3c520cde7\": rpc error: code = NotFound desc = could not find container \"989327640a7fc29aa47ea390815f9a931b57fc963cca4d80a8d6f4b3c520cde7\": container with ID starting with 989327640a7fc29aa47ea390815f9a931b57fc963cca4d80a8d6f4b3c520cde7 not found: ID does not exist" Oct 30 12:43:52.205520 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:52.205484 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm"] Oct 30 12:43:52.210174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:52.210140 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ac674b-79fb85957f-6zscm"] Oct 30 12:43:53.004238 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:53.004198 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:53.024852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:53.024817 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0816edc6-cc13-4ffb-87f8-8661aedfb0fe" path="/var/lib/kubelet/pods/0816edc6-cc13-4ffb-87f8-8661aedfb0fe/volumes" Oct 30 12:43:54.622144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:54.622112 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:56.398515 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:56.398481 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:58.352101 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:58.352067 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:43:59.934657 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:43:59.934623 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:00.130647 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:00.130589 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:00.137469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:00.137449 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" Oct 30 12:44:00.391004 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:00.390971 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:00.570617 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:00.570587 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:01.491724 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:01.491689 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:01.699680 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:01.699648 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:02.003695 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:02.003661 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:02.324788 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:02.324707 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:02.504580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:02.504546 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:02.779177 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:02.779143 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:03.072656 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:03.072579 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:03.295999 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:03.295944 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:04.149893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:04.149834 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:04.534361 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:04.534237 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:05.372928 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:05.372893 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:05.667995 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:05.667961 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:05.816788 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:44:05.816752 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:44:06.593917 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:06.593883 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:06.843666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:06.843629 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576"] Oct 30 12:44:06.844475 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:06.844431 4413 cpu_manager.go:414] "containerMap: removing container" podUID="0816edc6-cc13-4ffb-87f8-8661aedfb0fe" containerName="payload-pre-processing" Oct 30 12:44:06.844475 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:06.844448 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0816edc6-cc13-4ffb-87f8-8661aedfb0fe" containerName="payload-pre-processing" Oct 30 12:44:06.844591 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:06.844582 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="0816edc6-cc13-4ffb-87f8-8661aedfb0fe" containerName="payload-pre-processing" Oct 30 12:44:06.850054 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:06.850033 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" Oct 30 12:44:06.865360 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:06.865331 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"kube-root-ca.crt\"" Oct 30 12:44:06.868744 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:06.868713 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576"] Oct 30 12:44:06.876585 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:06.876561 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"openshift-service-ca.crt\"" Oct 30 12:44:06.876739 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:06.876677 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"routing-overlay\"" Oct 30 12:44:06.877358 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:06.877338 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0de4b9\"" Oct 30 12:44:06.877462 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:06.877430 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"agc-cred-d7c4ac25d5ef\"" Oct 30 12:44:06.877569 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:06.877554 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"payload-processing-external-model-e2e--d115ed06-dockercfg-4k7rq\"" Oct 30 12:44:06.916540 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:06.916506 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd33196-cd20-4832-9cb0-53063fb91c11-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576\" (UID: \"4cd33196-cd20-4832-9cb0-53063fb91c11\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" Oct 30 12:44:06.916712 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:06.916569 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4cd33196-cd20-4832-9cb0-53063fb91c11-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576\" (UID: \"4cd33196-cd20-4832-9cb0-53063fb91c11\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" Oct 30 12:44:06.916712 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:06.916680 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csx45\" (UniqueName: \"kubernetes.io/projected/4cd33196-cd20-4832-9cb0-53063fb91c11-kube-api-access-csx45\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576\" (UID: \"4cd33196-cd20-4832-9cb0-53063fb91c11\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" Oct 30 12:44:06.916797 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:06.916717 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4cd33196-cd20-4832-9cb0-53063fb91c11-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576\" (UID: \"4cd33196-cd20-4832-9cb0-53063fb91c11\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" Oct 30 12:44:07.020764 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:07.020547 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd33196-cd20-4832-9cb0-53063fb91c11-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576\" (UID: \"4cd33196-cd20-4832-9cb0-53063fb91c11\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" Oct 30 12:44:07.020764 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:07.020613 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4cd33196-cd20-4832-9cb0-53063fb91c11-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576\" (UID: \"4cd33196-cd20-4832-9cb0-53063fb91c11\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" Oct 30 12:44:07.020764 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:07.020699 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csx45\" (UniqueName: \"kubernetes.io/projected/4cd33196-cd20-4832-9cb0-53063fb91c11-kube-api-access-csx45\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576\" (UID: \"4cd33196-cd20-4832-9cb0-53063fb91c11\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" Oct 30 12:44:07.020764 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:07.020749 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4cd33196-cd20-4832-9cb0-53063fb91c11-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576\" (UID: \"4cd33196-cd20-4832-9cb0-53063fb91c11\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" Oct 30 12:44:07.021843 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:07.021597 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4cd33196-cd20-4832-9cb0-53063fb91c11-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576\" (UID: \"4cd33196-cd20-4832-9cb0-53063fb91c11\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" Oct 30 12:44:07.022365 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:07.022340 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd33196-cd20-4832-9cb0-53063fb91c11-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576\" (UID: \"4cd33196-cd20-4832-9cb0-53063fb91c11\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" Oct 30 12:44:07.050520 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:07.050432 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4cd33196-cd20-4832-9cb0-53063fb91c11-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576\" (UID: \"4cd33196-cd20-4832-9cb0-53063fb91c11\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" Oct 30 12:44:07.066844 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:07.066655 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csx45\" (UniqueName: \"kubernetes.io/projected/4cd33196-cd20-4832-9cb0-53063fb91c11-kube-api-access-csx45\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576\" (UID: \"4cd33196-cd20-4832-9cb0-53063fb91c11\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" Oct 30 12:44:07.179141 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:07.179113 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"payload-processing-external-model-e2e--d115ed06-dockercfg-4k7rq\"" Oct 30 12:44:07.186605 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:07.186582 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" Oct 30 12:44:07.437072 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:07.437003 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:07.507929 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:07.507903 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576"] Oct 30 12:44:07.508557 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:44:07.508529 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd33196_cd20_4832_9cb0_53063fb91c11.slice/crio-7cc9e55b0cdf0144deb50abad9b974b585078253af03bf69cf30107c4788bc4a WatchSource:0}: Error finding container 7cc9e55b0cdf0144deb50abad9b974b585078253af03bf69cf30107c4788bc4a: Status 404 returned error can't find the container with id 7cc9e55b0cdf0144deb50abad9b974b585078253af03bf69cf30107c4788bc4a Oct 30 12:44:08.288111 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:08.288079 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" event={"ID":"4cd33196-cd20-4832-9cb0-53063fb91c11","Type":"ContainerStarted","Data":"ce3c53632f047f36e60ff49ff03d729b2746367aaa3fababde2ead60f268e7d9"} Oct 30 12:44:08.288531 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:08.288134 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" event={"ID":"4cd33196-cd20-4832-9cb0-53063fb91c11","Type":"ContainerStarted","Data":"7cc9e55b0cdf0144deb50abad9b974b585078253af03bf69cf30107c4788bc4a"} Oct 30 12:44:08.288531 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:08.288288 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" Oct 30 12:44:08.335958 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:08.335911 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" podStartSLOduration=2.335897867 podStartE2EDuration="2.335897867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:08.331956164 +0000 UTC m=+3001.441799036" watchObservedRunningTime="2026-10-30 12:44:08.335897867 +0000 UTC m=+3001.445740747" Oct 30 12:44:08.818822 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:08.818793 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:10.436820 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:10.436777 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:11.865933 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:11.865898 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:12.185135 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:12.185104 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:13.959442 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:13.959387 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:15.643680 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:15.643639 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:17.440065 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:17.440030 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:19.241669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:19.241455 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:19.298388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:19.298359 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" Oct 30 12:44:20.854262 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:20.854227 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:22.694854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:22.694825 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:24.497717 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:24.497678 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:26.393879 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:26.393845 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:28.180765 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:28.180735 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:30.160538 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:30.160509 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:31.915926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:31.915888 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n"] Oct 30 12:44:31.916543 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:31.916144 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" podUID="df7b3c5e-ca33-453d-b0a9-9adc5ea8840c" containerName="maas-api" containerID="cri-o://0c8002f4c633bfdb89b2bc382df9ca5d8fa232e15c46fed00c05029d3659d45f" gracePeriod=30 Oct 30 12:44:32.098061 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.097951 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:32.211655 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.211624 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" Oct 30 12:44:32.279576 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.279539 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-kube-api-access-vhpd5\" (UniqueName: \"kubernetes.io/projected/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-kube-api-access-vhpd5\") pod \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\" (UID: \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\") " Oct 30 12:44:32.279775 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.279743 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-maas-api-tls\") pod \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\" (UID: \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\") " Oct 30 12:44:32.279869 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.279851 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-metrics-certs\") pod \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\" (UID: \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\") " Oct 30 12:44:32.279924 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.279888 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-openshift-service-ca\") pod \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\" (UID: \"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c\") " Oct 30 12:44:32.280810 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.280777 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-openshift-service-ca" pod "df7b3c5e-ca33-453d-b0a9-9adc5ea8840c" (UID: "df7b3c5e-ca33-453d-b0a9-9adc5ea8840c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:44:32.282333 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.282300 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-kube-api-access-vhpd5" pod "df7b3c5e-ca33-453d-b0a9-9adc5ea8840c" (UID: "df7b3c5e-ca33-453d-b0a9-9adc5ea8840c"). InnerVolumeSpecName "kube-api-access-vhpd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:44:32.282537 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.282515 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-maas-api-tls" pod "df7b3c5e-ca33-453d-b0a9-9adc5ea8840c" (UID: "df7b3c5e-ca33-453d-b0a9-9adc5ea8840c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:44:32.282814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.282790 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-metrics-certs" pod "df7b3c5e-ca33-453d-b0a9-9adc5ea8840c" (UID: "df7b3c5e-ca33-453d-b0a9-9adc5ea8840c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 12:44:32.382362 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.382334 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:44:32.382362 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.382357 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:44:32.382362 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.382366 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:44:32.382603 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.382384 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhpd5\" (UniqueName: \"kubernetes.io/projected/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c-kube-api-access-vhpd5\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:44:32.462381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.462288 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:32.476193 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.476153 4413 generic.go:356] "Generic (PLEG): container finished" podID="df7b3c5e-ca33-453d-b0a9-9adc5ea8840c" containerID="0c8002f4c633bfdb89b2bc382df9ca5d8fa232e15c46fed00c05029d3659d45f" exitCode=0 Oct 30 12:44:32.476450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.476329 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" event={"ID":"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c","Type":"ContainerDied","Data":"0c8002f4c633bfdb89b2bc382df9ca5d8fa232e15c46fed00c05029d3659d45f"} Oct 30 12:44:32.476450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.476388 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" event={"ID":"df7b3c5e-ca33-453d-b0a9-9adc5ea8840c","Type":"ContainerDied","Data":"2247917f1ede61f11b6201ce7db0d26461a3c90ccd17a4e586ea48317e37369f"} Oct 30 12:44:32.477222 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.477190 4413 scope.go:122] "RemoveContainer" containerID="0c8002f4c633bfdb89b2bc382df9ca5d8fa232e15c46fed00c05029d3659d45f" Oct 30 12:44:32.477406 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.477390 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n" Oct 30 12:44:32.518322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.517995 4413 scope.go:122] "RemoveContainer" containerID="0c8002f4c633bfdb89b2bc382df9ca5d8fa232e15c46fed00c05029d3659d45f" Oct 30 12:44:32.524555 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:44:32.522096 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c8002f4c633bfdb89b2bc382df9ca5d8fa232e15c46fed00c05029d3659d45f\": container with ID starting with 0c8002f4c633bfdb89b2bc382df9ca5d8fa232e15c46fed00c05029d3659d45f not found: ID does not exist" containerID="0c8002f4c633bfdb89b2bc382df9ca5d8fa232e15c46fed00c05029d3659d45f" Oct 30 12:44:32.524555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.522139 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c8002f4c633bfdb89b2bc382df9ca5d8fa232e15c46fed00c05029d3659d45f"} err="failed to get container status \"0c8002f4c633bfdb89b2bc382df9ca5d8fa232e15c46fed00c05029d3659d45f\": rpc error: code = NotFound desc = could not find container \"0c8002f4c633bfdb89b2bc382df9ca5d8fa232e15c46fed00c05029d3659d45f\": container with ID starting with 0c8002f4c633bfdb89b2bc382df9ca5d8fa232e15c46fed00c05029d3659d45f not found: ID does not exist" Oct 30 12:44:32.554975 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.554940 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n"] Oct 30 12:44:32.559167 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.559145 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ac674b-b69698c54-pk74n"] Oct 30 12:44:32.799867 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.799774 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:32.989456 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:32.989425 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:33.021380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:33.021353 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df7b3c5e-ca33-453d-b0a9-9adc5ea8840c" path="/var/lib/kubelet/pods/df7b3c5e-ca33-453d-b0a9-9adc5ea8840c/volumes" Oct 30 12:44:33.388628 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:33.388600 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:33.808612 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:33.808536 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:34.024218 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:34.024187 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:34.196930 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:34.196900 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:34.490140 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:34.490039 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:34.717086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:34.717052 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:34.925105 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:34.925062 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:35.285190 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:35.285119 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:35.565570 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:35.565496 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:35.814876 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:44:35.814843 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:44:36.711850 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:36.711801 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:36.897532 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:36.897503 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:36.927875 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:36.927829 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr"] Oct 30 12:44:36.928697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:36.928676 4413 cpu_manager.go:414] "containerMap: removing container" podUID="df7b3c5e-ca33-453d-b0a9-9adc5ea8840c" containerName="maas-api" Oct 30 12:44:36.928697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:36.928697 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df7b3c5e-ca33-453d-b0a9-9adc5ea8840c" containerName="maas-api" Oct 30 12:44:36.928904 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:36.928889 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="df7b3c5e-ca33-453d-b0a9-9adc5ea8840c" containerName="maas-api" Oct 30 12:44:36.934024 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:36.934005 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" Oct 30 12:44:36.949099 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:36.948994 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ecb3aa\"/\"agc-cred-09e51c302db7\"" Oct 30 12:44:36.949564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:36.949546 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ecb3aa\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ecb3aa\"" Oct 30 12:44:36.967431 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:36.967347 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr"] Oct 30 12:44:37.049402 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:37.049369 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ecb3aa\"/\"openshift-service-ca.crt\"" Oct 30 12:44:37.049576 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:37.049423 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ecb3aa\"/\"routing-overlay\"" Oct 30 12:44:37.049648 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:37.049635 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ecb3aa\"/\"payload-processing-external-model-e2e--8849cfcf-dockercfg-kgh5v\"" Oct 30 12:44:37.049692 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:37.049653 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ecb3aa\"/\"kube-root-ca.crt\"" Oct 30 12:44:37.084272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:37.083210 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa60d02-3a82-4f79-9382-bc3a84180c08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr\" (UID: \"7fa60d02-3a82-4f79-9382-bc3a84180c08\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" Oct 30 12:44:37.084272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:37.083472 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fa60d02-3a82-4f79-9382-bc3a84180c08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr\" (UID: \"7fa60d02-3a82-4f79-9382-bc3a84180c08\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" Oct 30 12:44:37.084272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:37.083517 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fa60d02-3a82-4f79-9382-bc3a84180c08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr\" (UID: \"7fa60d02-3a82-4f79-9382-bc3a84180c08\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" Oct 30 12:44:37.084272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:37.083603 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsm6t\" (UniqueName: \"kubernetes.io/projected/7fa60d02-3a82-4f79-9382-bc3a84180c08-kube-api-access-qsm6t\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr\" (UID: \"7fa60d02-3a82-4f79-9382-bc3a84180c08\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" Oct 30 12:44:37.185187 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:37.185144 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fa60d02-3a82-4f79-9382-bc3a84180c08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr\" (UID: \"7fa60d02-3a82-4f79-9382-bc3a84180c08\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" Oct 30 12:44:37.185441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:37.185196 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fa60d02-3a82-4f79-9382-bc3a84180c08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr\" (UID: \"7fa60d02-3a82-4f79-9382-bc3a84180c08\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" Oct 30 12:44:37.185441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:37.185309 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsm6t\" (UniqueName: \"kubernetes.io/projected/7fa60d02-3a82-4f79-9382-bc3a84180c08-kube-api-access-qsm6t\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr\" (UID: \"7fa60d02-3a82-4f79-9382-bc3a84180c08\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" Oct 30 12:44:37.185571 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:37.185457 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa60d02-3a82-4f79-9382-bc3a84180c08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr\" (UID: \"7fa60d02-3a82-4f79-9382-bc3a84180c08\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" Oct 30 12:44:37.186203 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:37.186124 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fa60d02-3a82-4f79-9382-bc3a84180c08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr\" (UID: \"7fa60d02-3a82-4f79-9382-bc3a84180c08\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" Oct 30 12:44:37.186363 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:37.186237 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa60d02-3a82-4f79-9382-bc3a84180c08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr\" (UID: \"7fa60d02-3a82-4f79-9382-bc3a84180c08\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" Oct 30 12:44:37.188059 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:37.188036 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fa60d02-3a82-4f79-9382-bc3a84180c08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr\" (UID: \"7fa60d02-3a82-4f79-9382-bc3a84180c08\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" Oct 30 12:44:37.210941 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:37.210917 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsm6t\" (UniqueName: \"kubernetes.io/projected/7fa60d02-3a82-4f79-9382-bc3a84180c08-kube-api-access-qsm6t\") pod \"payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr\" (UID: \"7fa60d02-3a82-4f79-9382-bc3a84180c08\") " pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" Oct 30 12:44:37.261211 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:37.261164 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" Oct 30 12:44:37.570559 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:37.570531 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr"] Oct 30 12:44:37.571464 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:44:37.571440 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa60d02_3a82_4f79_9382_bc3a84180c08.slice/crio-ff3bc7332df0db5e341bdb5ef3ed3eb839080b8e5aadf9054877335d1c393f25 WatchSource:0}: Error finding container ff3bc7332df0db5e341bdb5ef3ed3eb839080b8e5aadf9054877335d1c393f25: Status 404 returned error can't find the container with id ff3bc7332df0db5e341bdb5ef3ed3eb839080b8e5aadf9054877335d1c393f25 Oct 30 12:44:38.523966 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:38.523924 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" event={"ID":"7fa60d02-3a82-4f79-9382-bc3a84180c08","Type":"ContainerStarted","Data":"a2b1027cc6cd3d31f9fba667369555c70f17053c99c6d5cc548a4be79a579760"} Oct 30 12:44:38.524432 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:38.524132 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" Oct 30 12:44:38.524432 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:38.524163 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" event={"ID":"7fa60d02-3a82-4f79-9382-bc3a84180c08","Type":"ContainerStarted","Data":"ff3bc7332df0db5e341bdb5ef3ed3eb839080b8e5aadf9054877335d1c393f25"} Oct 30 12:44:38.572438 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:38.572380 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" podStartSLOduration=2.572365624 podStartE2EDuration="2.572365624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:38.570488258 +0000 UTC m=+3031.680331140" watchObservedRunningTime="2026-10-30 12:44:38.572365624 +0000 UTC m=+3031.682208505" Oct 30 12:44:38.706186 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:38.706136 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:40.457019 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:40.456979 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:40.664094 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:40.664054 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:40.827748 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:40.827663 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:41.071263 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:41.071226 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:42.372225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:42.372194 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:42.946006 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:42.945967 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:43.472637 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:43.472591 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:43.937236 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:43.937208 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:44.474618 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:44.474585 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:46.332892 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:46.332854 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:47.880677 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:47.880639 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:48.050311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:48.050282 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:48.277821 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:48.277729 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:49.534686 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:49.534654 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" Oct 30 12:44:49.721378 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:49.721347 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:49.828215 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:49.828135 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv"] Oct 30 12:44:49.833299 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:49.833280 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv" Oct 30 12:44:49.851901 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:49.851871 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-10f364\"" Oct 30 12:44:49.852140 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:49.851925 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-10f364-dockercfg-zj9wb\"" Oct 30 12:44:49.852723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:49.852699 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv"] Oct 30 12:44:49.934493 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:49.934452 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae223397-5523-4836-9d6c-7e2daf2ae313-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv\" (UID: \"ae223397-5523-4836-9d6c-7e2daf2ae313\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv" Oct 30 12:44:49.935022 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:49.934994 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chfnw\" (UniqueName: \"kubernetes.io/projected/ae223397-5523-4836-9d6c-7e2daf2ae313-kube-api-access-chfnw\") pod \"payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv\" (UID: \"ae223397-5523-4836-9d6c-7e2daf2ae313\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv" Oct 30 12:44:49.951795 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:49.951756 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:50.037872 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.037634 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chfnw\" (UniqueName: \"kubernetes.io/projected/ae223397-5523-4836-9d6c-7e2daf2ae313-kube-api-access-chfnw\") pod \"payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv\" (UID: \"ae223397-5523-4836-9d6c-7e2daf2ae313\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv" Oct 30 12:44:50.038411 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.038369 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae223397-5523-4836-9d6c-7e2daf2ae313-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv\" (UID: \"ae223397-5523-4836-9d6c-7e2daf2ae313\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv" Oct 30 12:44:50.039634 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.039600 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae223397-5523-4836-9d6c-7e2daf2ae313-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv\" (UID: \"ae223397-5523-4836-9d6c-7e2daf2ae313\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv" Oct 30 12:44:50.040130 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.040101 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d"] Oct 30 12:44:50.051297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.051275 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d"] Oct 30 12:44:50.051531 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.051519 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d" Oct 30 12:44:50.075667 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.075640 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chfnw\" (UniqueName: \"kubernetes.io/projected/ae223397-5523-4836-9d6c-7e2daf2ae313-kube-api-access-chfnw\") pod \"payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv\" (UID: \"ae223397-5523-4836-9d6c-7e2daf2ae313\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv" Oct 30 12:44:50.140210 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.140172 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxw9q\" (UniqueName: \"kubernetes.io/projected/4295a82c-12f5-4451-b82a-0203044f1de3-kube-api-access-wxw9q\") pod \"payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d\" (UID: \"4295a82c-12f5-4451-b82a-0203044f1de3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d" Oct 30 12:44:50.140429 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.140404 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4295a82c-12f5-4451-b82a-0203044f1de3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d\" (UID: \"4295a82c-12f5-4451-b82a-0203044f1de3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d" Oct 30 12:44:50.149699 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.149678 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv" Oct 30 12:44:50.242117 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.242082 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxw9q\" (UniqueName: \"kubernetes.io/projected/4295a82c-12f5-4451-b82a-0203044f1de3-kube-api-access-wxw9q\") pod \"payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d\" (UID: \"4295a82c-12f5-4451-b82a-0203044f1de3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d" Oct 30 12:44:50.242331 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.242161 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4295a82c-12f5-4451-b82a-0203044f1de3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d\" (UID: \"4295a82c-12f5-4451-b82a-0203044f1de3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d" Oct 30 12:44:50.244095 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.243384 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4295a82c-12f5-4451-b82a-0203044f1de3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d\" (UID: \"4295a82c-12f5-4451-b82a-0203044f1de3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d" Oct 30 12:44:50.266482 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.266451 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxw9q\" (UniqueName: \"kubernetes.io/projected/4295a82c-12f5-4451-b82a-0203044f1de3-kube-api-access-wxw9q\") pod \"payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d\" (UID: \"4295a82c-12f5-4451-b82a-0203044f1de3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d" Oct 30 12:44:50.408622 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.408594 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d" Oct 30 12:44:50.467576 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.467327 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv"] Oct 30 12:44:50.500474 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:44:50.500421 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae223397_5523_4836_9d6c_7e2daf2ae313.slice/crio-4376109bd5e225ee463e1132ce21163ab53c0820452ca493b7abbe287e50f604 WatchSource:0}: Error finding container 4376109bd5e225ee463e1132ce21163ab53c0820452ca493b7abbe287e50f604: Status 404 returned error can't find the container with id 4376109bd5e225ee463e1132ce21163ab53c0820452ca493b7abbe287e50f604 Oct 30 12:44:50.501608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.501578 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:50.622958 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.622926 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv" event={"ID":"ae223397-5523-4836-9d6c-7e2daf2ae313","Type":"ContainerStarted","Data":"6ce3aa07d7c96003104c6fff246ec2596986c6b06b7fe4afe4ce36a5abbf4d39"} Oct 30 12:44:50.623396 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.622976 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv" event={"ID":"ae223397-5523-4836-9d6c-7e2daf2ae313","Type":"ContainerStarted","Data":"4376109bd5e225ee463e1132ce21163ab53c0820452ca493b7abbe287e50f604"} Oct 30 12:44:50.623396 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.623008 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv" Oct 30 12:44:50.668828 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.668713 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv" podStartSLOduration=1.668698381 podStartE2EDuration="1.668698381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:50.665030835 +0000 UTC m=+3043.774873716" watchObservedRunningTime="2026-10-30 12:44:50.668698381 +0000 UTC m=+3043.778541282" Oct 30 12:44:50.771358 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:44:50.771326 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4295a82c_12f5_4451_b82a_0203044f1de3.slice/crio-a352db8ff5bf03c3b383aa546c9951a9b96abdfa439b3942e6646aa3a8a2cdcb WatchSource:0}: Error finding container a352db8ff5bf03c3b383aa546c9951a9b96abdfa439b3942e6646aa3a8a2cdcb: Status 404 returned error can't find the container with id a352db8ff5bf03c3b383aa546c9951a9b96abdfa439b3942e6646aa3a8a2cdcb Oct 30 12:44:50.771485 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:50.771428 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d"] Oct 30 12:44:51.631605 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:51.631571 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d" event={"ID":"4295a82c-12f5-4451-b82a-0203044f1de3","Type":"ContainerStarted","Data":"fa5c880995796b92034c6b319bc660b4b1d6bec9bc52c5e4cf5d1da08642ecc3"} Oct 30 12:44:51.632047 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:51.631657 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d" event={"ID":"4295a82c-12f5-4451-b82a-0203044f1de3","Type":"ContainerStarted","Data":"a352db8ff5bf03c3b383aa546c9951a9b96abdfa439b3942e6646aa3a8a2cdcb"} Oct 30 12:44:51.632047 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:51.631824 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d" Oct 30 12:44:51.677571 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:51.677520 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d" podStartSLOduration=2.67750673 podStartE2EDuration="2.67750673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:51.672585804 +0000 UTC m=+3044.782428686" watchObservedRunningTime="2026-10-30 12:44:51.67750673 +0000 UTC m=+3044.787349611" Oct 30 12:44:52.558332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:52.558297 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:54.567639 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:54.567608 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:56.460308 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:56.460274 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:44:58.206293 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:44:58.206264 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:00.045431 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.045400 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:00.261672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.261636 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:00.329417 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.329318 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n"] Oct 30 12:45:00.334846 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.334823 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" Oct 30 12:45:00.374037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.374001 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7"] Oct 30 12:45:00.378864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.378838 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" Oct 30 12:45:00.391758 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.391725 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4b98ce-81d9-4aff-9c27-7c27119fc440-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n\" (UID: \"7d4b98ce-81d9-4aff-9c27-7c27119fc440\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" Oct 30 12:45:00.391901 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.391820 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb3d669-0630-425d-926f-fdfedac04d40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7\" (UID: \"9cb3d669-0630-425d-926f-fdfedac04d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" Oct 30 12:45:00.391901 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.391863 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc9mh\" (UniqueName: \"kubernetes.io/projected/9cb3d669-0630-425d-926f-fdfedac04d40-kube-api-access-lc9mh\") pod \"maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7\" (UID: \"9cb3d669-0630-425d-926f-fdfedac04d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" Oct 30 12:45:00.391901 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.391896 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx2tq\" (UniqueName: \"kubernetes.io/projected/7d4b98ce-81d9-4aff-9c27-7c27119fc440-kube-api-access-xx2tq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n\" (UID: \"7d4b98ce-81d9-4aff-9c27-7c27119fc440\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" Oct 30 12:45:00.453489 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.453456 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 12:45:00.453703 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.453682 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm"] Oct 30 12:45:00.472825 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.472783 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7"] Oct 30 12:45:00.472996 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.472895 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" Oct 30 12:45:00.491176 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.491140 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm"] Oct 30 12:45:00.493636 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.493581 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4b98ce-81d9-4aff-9c27-7c27119fc440-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n\" (UID: \"7d4b98ce-81d9-4aff-9c27-7c27119fc440\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" Oct 30 12:45:00.493840 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.493706 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb3d669-0630-425d-926f-fdfedac04d40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7\" (UID: \"9cb3d669-0630-425d-926f-fdfedac04d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" Oct 30 12:45:00.493840 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.493750 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc9mh\" (UniqueName: \"kubernetes.io/projected/9cb3d669-0630-425d-926f-fdfedac04d40-kube-api-access-lc9mh\") pod \"maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7\" (UID: \"9cb3d669-0630-425d-926f-fdfedac04d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" Oct 30 12:45:00.493840 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.493790 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx2tq\" (UniqueName: \"kubernetes.io/projected/7d4b98ce-81d9-4aff-9c27-7c27119fc440-kube-api-access-xx2tq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n\" (UID: \"7d4b98ce-81d9-4aff-9c27-7c27119fc440\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" Oct 30 12:45:00.494021 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.493847 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/614cb5f4-97b0-46f2-8d23-c50b5b673eab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm\" (UID: \"614cb5f4-97b0-46f2-8d23-c50b5b673eab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" Oct 30 12:45:00.494021 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.493899 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvt4d\" (UniqueName: \"kubernetes.io/projected/614cb5f4-97b0-46f2-8d23-c50b5b673eab-kube-api-access-gvt4d\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm\" (UID: \"614cb5f4-97b0-46f2-8d23-c50b5b673eab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" Oct 30 12:45:00.494506 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.494478 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4b98ce-81d9-4aff-9c27-7c27119fc440-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n\" (UID: \"7d4b98ce-81d9-4aff-9c27-7c27119fc440\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" Oct 30 12:45:00.494714 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.494694 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb3d669-0630-425d-926f-fdfedac04d40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7\" (UID: \"9cb3d669-0630-425d-926f-fdfedac04d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" Oct 30 12:45:00.496190 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.496167 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj"] Oct 30 12:45:00.501121 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.501103 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" Oct 30 12:45:00.529582 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.527998 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n"] Oct 30 12:45:00.532569 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.532550 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj"] Oct 30 12:45:00.535997 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.535977 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx2tq\" (UniqueName: \"kubernetes.io/projected/7d4b98ce-81d9-4aff-9c27-7c27119fc440-kube-api-access-xx2tq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n\" (UID: \"7d4b98ce-81d9-4aff-9c27-7c27119fc440\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" Oct 30 12:45:00.550837 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.550793 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc9mh\" (UniqueName: \"kubernetes.io/projected/9cb3d669-0630-425d-926f-fdfedac04d40-kube-api-access-lc9mh\") pod \"maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7\" (UID: \"9cb3d669-0630-425d-926f-fdfedac04d40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" Oct 30 12:45:00.595048 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.594952 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/614cb5f4-97b0-46f2-8d23-c50b5b673eab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm\" (UID: \"614cb5f4-97b0-46f2-8d23-c50b5b673eab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" Oct 30 12:45:00.595048 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.595039 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvt4d\" (UniqueName: \"kubernetes.io/projected/614cb5f4-97b0-46f2-8d23-c50b5b673eab-kube-api-access-gvt4d\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm\" (UID: \"614cb5f4-97b0-46f2-8d23-c50b5b673eab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" Oct 30 12:45:00.595307 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.595281 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b4a2cf-88e0-43af-a20d-f8d79f76368c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889405-jhtjj\" (UID: \"e3b4a2cf-88e0-43af-a20d-f8d79f76368c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" Oct 30 12:45:00.595375 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.595334 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbwp4\" (UniqueName: \"kubernetes.io/projected/e3b4a2cf-88e0-43af-a20d-f8d79f76368c-kube-api-access-jbwp4\") pod \"maas-api-key-cleanup-29889405-jhtjj\" (UID: \"e3b4a2cf-88e0-43af-a20d-f8d79f76368c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" Oct 30 12:45:00.595764 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.595743 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/614cb5f4-97b0-46f2-8d23-c50b5b673eab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm\" (UID: \"614cb5f4-97b0-46f2-8d23-c50b5b673eab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" Oct 30 12:45:00.636494 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.636461 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvt4d\" (UniqueName: \"kubernetes.io/projected/614cb5f4-97b0-46f2-8d23-c50b5b673eab-kube-api-access-gvt4d\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm\" (UID: \"614cb5f4-97b0-46f2-8d23-c50b5b673eab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" Oct 30 12:45:00.655867 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.655838 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" Oct 30 12:45:00.694367 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.694341 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" Oct 30 12:45:00.696640 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.696610 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b4a2cf-88e0-43af-a20d-f8d79f76368c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889405-jhtjj\" (UID: \"e3b4a2cf-88e0-43af-a20d-f8d79f76368c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" Oct 30 12:45:00.696827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.696660 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbwp4\" (UniqueName: \"kubernetes.io/projected/e3b4a2cf-88e0-43af-a20d-f8d79f76368c-kube-api-access-jbwp4\") pod \"maas-api-key-cleanup-29889405-jhtjj\" (UID: \"e3b4a2cf-88e0-43af-a20d-f8d79f76368c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" Oct 30 12:45:00.697355 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.697323 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b4a2cf-88e0-43af-a20d-f8d79f76368c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889405-jhtjj\" (UID: \"e3b4a2cf-88e0-43af-a20d-f8d79f76368c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" Oct 30 12:45:00.726409 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.726375 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbwp4\" (UniqueName: \"kubernetes.io/projected/e3b4a2cf-88e0-43af-a20d-f8d79f76368c-kube-api-access-jbwp4\") pod \"maas-api-key-cleanup-29889405-jhtjj\" (UID: \"e3b4a2cf-88e0-43af-a20d-f8d79f76368c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" Oct 30 12:45:00.795088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.794947 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" Oct 30 12:45:00.815362 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.815092 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" Oct 30 12:45:00.988758 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:00.988467 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n"] Oct 30 12:45:00.990668 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:45:00.990640 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4b98ce_81d9_4aff_9c27_7c27119fc440.slice/crio-a223e58bfa21b0e853d6e2a9a3925265f65c6b61e4009614fe45abaeee7f7529 WatchSource:0}: Error finding container a223e58bfa21b0e853d6e2a9a3925265f65c6b61e4009614fe45abaeee7f7529: Status 404 returned error can't find the container with id a223e58bfa21b0e853d6e2a9a3925265f65c6b61e4009614fe45abaeee7f7529 Oct 30 12:45:01.047313 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:45:01.047281 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb3d669_0630_425d_926f_fdfedac04d40.slice/crio-2850a3a169d1413f9ad9b1c37f522fa9b4c1f785c44fa97b6f573ec6a50d6bf2 WatchSource:0}: Error finding container 2850a3a169d1413f9ad9b1c37f522fa9b4c1f785c44fa97b6f573ec6a50d6bf2: Status 404 returned error can't find the container with id 2850a3a169d1413f9ad9b1c37f522fa9b4c1f785c44fa97b6f573ec6a50d6bf2 Oct 30 12:45:01.047685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:01.047664 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7"] Oct 30 12:45:01.146538 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:01.146509 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm"] Oct 30 12:45:01.164240 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:01.164087 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj"] Oct 30 12:45:01.195449 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:45:01.195396 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614cb5f4_97b0_46f2_8d23_c50b5b673eab.slice/crio-0a7d617acac6144213e920cf64933f4bf66b125af8be568026cd316e2976169e WatchSource:0}: Error finding container 0a7d617acac6144213e920cf64933f4bf66b125af8be568026cd316e2976169e: Status 404 returned error can't find the container with id 0a7d617acac6144213e920cf64933f4bf66b125af8be568026cd316e2976169e Oct 30 12:45:01.195781 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:45:01.195758 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b4a2cf_88e0_43af_a20d_f8d79f76368c.slice/crio-9a8ef2e8f01f82a8467b080065a812bc8a44dd4e2deb7a0c1659747025415ee0 WatchSource:0}: Error finding container 9a8ef2e8f01f82a8467b080065a812bc8a44dd4e2deb7a0c1659747025415ee0: Status 404 returned error can't find the container with id 9a8ef2e8f01f82a8467b080065a812bc8a44dd4e2deb7a0c1659747025415ee0 Oct 30 12:45:01.634604 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:01.634579 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv" Oct 30 12:45:01.707793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:01.707694 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" event={"ID":"e3b4a2cf-88e0-43af-a20d-f8d79f76368c","Type":"ContainerStarted","Data":"8ba9452a195890dcce63ea1e6ab2fa08dfeae69e3901568127d89c8065975b98"} Oct 30 12:45:01.707793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:01.707763 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" event={"ID":"e3b4a2cf-88e0-43af-a20d-f8d79f76368c","Type":"ContainerStarted","Data":"9a8ef2e8f01f82a8467b080065a812bc8a44dd4e2deb7a0c1659747025415ee0"} Oct 30 12:45:01.711002 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:01.710965 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" event={"ID":"7d4b98ce-81d9-4aff-9c27-7c27119fc440","Type":"ContainerStarted","Data":"5ac2b2f91343836106d24c25e514799dcce181518dd286b62000f61ae17c1c89"} Oct 30 12:45:01.711171 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:01.711027 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" event={"ID":"7d4b98ce-81d9-4aff-9c27-7c27119fc440","Type":"ContainerStarted","Data":"a223e58bfa21b0e853d6e2a9a3925265f65c6b61e4009614fe45abaeee7f7529"} Oct 30 12:45:01.713478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:01.713452 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" event={"ID":"614cb5f4-97b0-46f2-8d23-c50b5b673eab","Type":"ContainerStarted","Data":"0a7d617acac6144213e920cf64933f4bf66b125af8be568026cd316e2976169e"} Oct 30 12:45:01.717381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:01.717354 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" event={"ID":"9cb3d669-0630-425d-926f-fdfedac04d40","Type":"ContainerStarted","Data":"0ea557ae36433c78055cc6fc94e48ccfdc44db85539514020d79efb120bdaa6e"} Oct 30 12:45:01.717495 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:01.717406 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" event={"ID":"9cb3d669-0630-425d-926f-fdfedac04d40","Type":"ContainerStarted","Data":"2850a3a169d1413f9ad9b1c37f522fa9b4c1f785c44fa97b6f573ec6a50d6bf2"} Oct 30 12:45:01.760843 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:01.760786 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" podStartSLOduration=1.760768018 podStartE2EDuration="1.760768018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:01.756101268 +0000 UTC m=+3054.865944169" watchObservedRunningTime="2026-10-30 12:45:01.760768018 +0000 UTC m=+3054.870610899" Oct 30 12:45:01.805525 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:01.805467 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" podStartSLOduration=1.805451141 podStartE2EDuration="1.805451141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:01.800159286 +0000 UTC m=+3054.910002167" watchObservedRunningTime="2026-10-30 12:45:01.805451141 +0000 UTC m=+3054.915294022" Oct 30 12:45:01.848148 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:01.848091 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" podStartSLOduration=1.848077237 podStartE2EDuration="1.848077237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:01.844556459 +0000 UTC m=+3054.954399340" watchObservedRunningTime="2026-10-30 12:45:01.848077237 +0000 UTC m=+3054.957920117" Oct 30 12:45:02.383757 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:02.383725 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:02.642480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:02.642446 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d" Oct 30 12:45:02.729416 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:02.729379 4413 generic.go:356] "Generic (PLEG): container finished" podID="614cb5f4-97b0-46f2-8d23-c50b5b673eab" containerID="609edbc15ded9a40827291608313f0651977dcd1944ef226d600f4b34b516a47" exitCode=60 Oct 30 12:45:02.729677 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:02.729512 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" event={"ID":"614cb5f4-97b0-46f2-8d23-c50b5b673eab","Type":"ContainerDied","Data":"609edbc15ded9a40827291608313f0651977dcd1944ef226d600f4b34b516a47"} Oct 30 12:45:02.729897 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:02.729873 4413 scope.go:122] "RemoveContainer" containerID="609edbc15ded9a40827291608313f0651977dcd1944ef226d600f4b34b516a47" Oct 30 12:45:02.734322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:02.734294 4413 generic.go:356] "Generic (PLEG): container finished" podID="9cb3d669-0630-425d-926f-fdfedac04d40" containerID="0ea557ae36433c78055cc6fc94e48ccfdc44db85539514020d79efb120bdaa6e" exitCode=60 Oct 30 12:45:02.734926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:02.734785 4413 scope.go:122] "RemoveContainer" containerID="0ea557ae36433c78055cc6fc94e48ccfdc44db85539514020d79efb120bdaa6e" Oct 30 12:45:02.734926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:02.734824 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" event={"ID":"9cb3d669-0630-425d-926f-fdfedac04d40","Type":"ContainerDied","Data":"0ea557ae36433c78055cc6fc94e48ccfdc44db85539514020d79efb120bdaa6e"} Oct 30 12:45:02.739863 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:02.737971 4413 generic.go:356] "Generic (PLEG): container finished" podID="e3b4a2cf-88e0-43af-a20d-f8d79f76368c" containerID="8ba9452a195890dcce63ea1e6ab2fa08dfeae69e3901568127d89c8065975b98" exitCode=60 Oct 30 12:45:02.739863 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:02.738124 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" event={"ID":"e3b4a2cf-88e0-43af-a20d-f8d79f76368c","Type":"ContainerDied","Data":"8ba9452a195890dcce63ea1e6ab2fa08dfeae69e3901568127d89c8065975b98"} Oct 30 12:45:02.739863 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:02.738835 4413 scope.go:122] "RemoveContainer" containerID="8ba9452a195890dcce63ea1e6ab2fa08dfeae69e3901568127d89c8065975b98" Oct 30 12:45:02.743332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:02.742223 4413 generic.go:356] "Generic (PLEG): container finished" podID="7d4b98ce-81d9-4aff-9c27-7c27119fc440" containerID="5ac2b2f91343836106d24c25e514799dcce181518dd286b62000f61ae17c1c89" exitCode=60 Oct 30 12:45:02.743332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:02.742286 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" event={"ID":"7d4b98ce-81d9-4aff-9c27-7c27119fc440","Type":"ContainerDied","Data":"5ac2b2f91343836106d24c25e514799dcce181518dd286b62000f61ae17c1c89"} Oct 30 12:45:02.743332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:02.742681 4413 scope.go:122] "RemoveContainer" containerID="5ac2b2f91343836106d24c25e514799dcce181518dd286b62000f61ae17c1c89" Oct 30 12:45:03.751850 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:03.751820 4413 generic.go:356] "Generic (PLEG): container finished" podID="e3b4a2cf-88e0-43af-a20d-f8d79f76368c" containerID="b51791aff9ad3a2908134b74cc7106f5ad3dffbe6420a1ca917fa866365f12c2" exitCode=60 Oct 30 12:45:03.752388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:03.751913 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" event={"ID":"e3b4a2cf-88e0-43af-a20d-f8d79f76368c","Type":"ContainerDied","Data":"b51791aff9ad3a2908134b74cc7106f5ad3dffbe6420a1ca917fa866365f12c2"} Oct 30 12:45:03.752388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:03.751981 4413 scope.go:122] "RemoveContainer" containerID="8ba9452a195890dcce63ea1e6ab2fa08dfeae69e3901568127d89c8065975b98" Oct 30 12:45:03.752388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:03.752269 4413 scope.go:122] "RemoveContainer" containerID="b51791aff9ad3a2908134b74cc7106f5ad3dffbe6420a1ca917fa866365f12c2" Oct 30 12:45:03.752663 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:45:03.752637 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889405-jhtjj_odh-ai-gateway-infra(e3b4a2cf-88e0-43af-a20d-f8d79f76368c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" podUID="e3b4a2cf-88e0-43af-a20d-f8d79f76368c" Oct 30 12:45:03.754325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:03.754298 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" event={"ID":"7d4b98ce-81d9-4aff-9c27-7c27119fc440","Type":"ContainerStarted","Data":"ee3cffb4c110d55a0096716827e97b59eb8bbb6318e928ce996218a27dc8fc44"} Oct 30 12:45:03.756130 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:03.756108 4413 generic.go:356] "Generic (PLEG): container finished" podID="614cb5f4-97b0-46f2-8d23-c50b5b673eab" containerID="67e3658fedf3ea8954df8a9ec6614e0113eed625af1ee24ade5f27609efaff89" exitCode=60 Oct 30 12:45:03.756222 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:03.756188 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" event={"ID":"614cb5f4-97b0-46f2-8d23-c50b5b673eab","Type":"ContainerDied","Data":"67e3658fedf3ea8954df8a9ec6614e0113eed625af1ee24ade5f27609efaff89"} Oct 30 12:45:03.756492 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:03.756474 4413 scope.go:122] "RemoveContainer" containerID="67e3658fedf3ea8954df8a9ec6614e0113eed625af1ee24ade5f27609efaff89" Oct 30 12:45:03.756799 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:45:03.756778 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm_odh-ai-gateway-infra(614cb5f4-97b0-46f2-8d23-c50b5b673eab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" podUID="614cb5f4-97b0-46f2-8d23-c50b5b673eab" Oct 30 12:45:03.758226 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:03.758201 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" event={"ID":"9cb3d669-0630-425d-926f-fdfedac04d40","Type":"ContainerStarted","Data":"8c4a70ad0b4e12f3cbdb34782c0537cbddd5206871a31733a7b6dd4557e6d469"} Oct 30 12:45:03.890848 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:03.890813 4413 scope.go:122] "RemoveContainer" containerID="609edbc15ded9a40827291608313f0651977dcd1944ef226d600f4b34b516a47" Oct 30 12:45:04.608605 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:04.608547 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:04.768333 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:04.768302 4413 scope.go:122] "RemoveContainer" containerID="b51791aff9ad3a2908134b74cc7106f5ad3dffbe6420a1ca917fa866365f12c2" Oct 30 12:45:04.768813 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:45:04.768762 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889405-jhtjj_odh-ai-gateway-infra(e3b4a2cf-88e0-43af-a20d-f8d79f76368c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" podUID="e3b4a2cf-88e0-43af-a20d-f8d79f76368c" Oct 30 12:45:04.770139 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:04.770110 4413 generic.go:356] "Generic (PLEG): container finished" podID="7d4b98ce-81d9-4aff-9c27-7c27119fc440" containerID="ee3cffb4c110d55a0096716827e97b59eb8bbb6318e928ce996218a27dc8fc44" exitCode=60 Oct 30 12:45:04.770311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:04.770187 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" event={"ID":"7d4b98ce-81d9-4aff-9c27-7c27119fc440","Type":"ContainerDied","Data":"ee3cffb4c110d55a0096716827e97b59eb8bbb6318e928ce996218a27dc8fc44"} Oct 30 12:45:04.770311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:04.770275 4413 scope.go:122] "RemoveContainer" containerID="5ac2b2f91343836106d24c25e514799dcce181518dd286b62000f61ae17c1c89" Oct 30 12:45:04.770559 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:04.770536 4413 scope.go:122] "RemoveContainer" containerID="ee3cffb4c110d55a0096716827e97b59eb8bbb6318e928ce996218a27dc8fc44" Oct 30 12:45:04.770937 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:45:04.770912 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n_odh-ai-gateway-infra(7d4b98ce-81d9-4aff-9c27-7c27119fc440)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" podUID="7d4b98ce-81d9-4aff-9c27-7c27119fc440" Oct 30 12:45:04.772594 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:04.772574 4413 scope.go:122] "RemoveContainer" containerID="67e3658fedf3ea8954df8a9ec6614e0113eed625af1ee24ade5f27609efaff89" Oct 30 12:45:04.772912 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:45:04.772889 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm_odh-ai-gateway-infra(614cb5f4-97b0-46f2-8d23-c50b5b673eab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" podUID="614cb5f4-97b0-46f2-8d23-c50b5b673eab" Oct 30 12:45:04.774284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:04.774233 4413 generic.go:356] "Generic (PLEG): container finished" podID="9cb3d669-0630-425d-926f-fdfedac04d40" containerID="8c4a70ad0b4e12f3cbdb34782c0537cbddd5206871a31733a7b6dd4557e6d469" exitCode=60 Oct 30 12:45:04.774385 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:04.774280 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" event={"ID":"9cb3d669-0630-425d-926f-fdfedac04d40","Type":"ContainerDied","Data":"8c4a70ad0b4e12f3cbdb34782c0537cbddd5206871a31733a7b6dd4557e6d469"} Oct 30 12:45:04.774559 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:04.774547 4413 scope.go:122] "RemoveContainer" containerID="8c4a70ad0b4e12f3cbdb34782c0537cbddd5206871a31733a7b6dd4557e6d469" Oct 30 12:45:04.774815 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:45:04.774797 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7_odh-ai-gateway-infra(9cb3d669-0630-425d-926f-fdfedac04d40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" podUID="9cb3d669-0630-425d-926f-fdfedac04d40" Oct 30 12:45:04.785242 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:04.785218 4413 scope.go:122] "RemoveContainer" containerID="0ea557ae36433c78055cc6fc94e48ccfdc44db85539514020d79efb120bdaa6e" Oct 30 12:45:05.789653 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:05.789608 4413 scope.go:122] "RemoveContainer" containerID="8c4a70ad0b4e12f3cbdb34782c0537cbddd5206871a31733a7b6dd4557e6d469" Oct 30 12:45:05.790102 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:45:05.790072 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7_odh-ai-gateway-infra(9cb3d669-0630-425d-926f-fdfedac04d40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" podUID="9cb3d669-0630-425d-926f-fdfedac04d40" Oct 30 12:45:05.792013 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:05.791994 4413 scope.go:122] "RemoveContainer" containerID="ee3cffb4c110d55a0096716827e97b59eb8bbb6318e928ce996218a27dc8fc44" Oct 30 12:45:05.792399 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:45:05.792376 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n_odh-ai-gateway-infra(7d4b98ce-81d9-4aff-9c27-7c27119fc440)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" podUID="7d4b98ce-81d9-4aff-9c27-7c27119fc440" Oct 30 12:45:05.819243 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:45:05.819205 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:45:06.350317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:06.350286 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:08.310569 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:08.310535 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:10.080742 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:10.080707 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:11.756805 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:11.756764 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:13.480867 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:13.480828 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:15.012517 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:15.012484 4413 scope.go:122] "RemoveContainer" containerID="ee3cffb4c110d55a0096716827e97b59eb8bbb6318e928ce996218a27dc8fc44" Oct 30 12:45:15.457585 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:15.457554 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:15.873030 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:15.872938 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" event={"ID":"7d4b98ce-81d9-4aff-9c27-7c27119fc440","Type":"ContainerStarted","Data":"5c1c7225e899e19822b3dd4c49989a36ba5666005b1fb6c7d6923e83df8f5ffb"} Oct 30 12:45:16.012155 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:16.012128 4413 scope.go:122] "RemoveContainer" containerID="67e3658fedf3ea8954df8a9ec6614e0113eed625af1ee24ade5f27609efaff89" Oct 30 12:45:16.882748 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:16.882706 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" event={"ID":"614cb5f4-97b0-46f2-8d23-c50b5b673eab","Type":"ContainerStarted","Data":"7aedc6a6daa5e5ef4db82de32b3e29c9f11bc5adba56a71b719a84f53edaa27f"} Oct 30 12:45:16.884752 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:16.884727 4413 generic.go:356] "Generic (PLEG): container finished" podID="7d4b98ce-81d9-4aff-9c27-7c27119fc440" containerID="5c1c7225e899e19822b3dd4c49989a36ba5666005b1fb6c7d6923e83df8f5ffb" exitCode=60 Oct 30 12:45:16.884865 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:16.884800 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" event={"ID":"7d4b98ce-81d9-4aff-9c27-7c27119fc440","Type":"ContainerDied","Data":"5c1c7225e899e19822b3dd4c49989a36ba5666005b1fb6c7d6923e83df8f5ffb"} Oct 30 12:45:16.884865 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:16.884860 4413 scope.go:122] "RemoveContainer" containerID="ee3cffb4c110d55a0096716827e97b59eb8bbb6318e928ce996218a27dc8fc44" Oct 30 12:45:16.885196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:16.885179 4413 scope.go:122] "RemoveContainer" containerID="5c1c7225e899e19822b3dd4c49989a36ba5666005b1fb6c7d6923e83df8f5ffb" Oct 30 12:45:16.885568 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:45:16.885545 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n_odh-ai-gateway-infra(7d4b98ce-81d9-4aff-9c27-7c27119fc440)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" podUID="7d4b98ce-81d9-4aff-9c27-7c27119fc440" Oct 30 12:45:16.926015 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:16.925953 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" podStartSLOduration=16.925939737 podStartE2EDuration="16.925939737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:16.922490186 +0000 UTC m=+3070.032333084" watchObservedRunningTime="2026-10-30 12:45:16.925939737 +0000 UTC m=+3070.035782617" Oct 30 12:45:16.936031 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:16.935987 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n"] Oct 30 12:45:17.018545 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:17.018518 4413 scope.go:122] "RemoveContainer" containerID="b51791aff9ad3a2908134b74cc7106f5ad3dffbe6420a1ca917fa866365f12c2" Oct 30 12:45:17.105993 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:17.105959 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:17.895208 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:17.895170 4413 generic.go:356] "Generic (PLEG): container finished" podID="614cb5f4-97b0-46f2-8d23-c50b5b673eab" containerID="7aedc6a6daa5e5ef4db82de32b3e29c9f11bc5adba56a71b719a84f53edaa27f" exitCode=60 Oct 30 12:45:17.895208 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:17.895198 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" event={"ID":"614cb5f4-97b0-46f2-8d23-c50b5b673eab","Type":"ContainerDied","Data":"7aedc6a6daa5e5ef4db82de32b3e29c9f11bc5adba56a71b719a84f53edaa27f"} Oct 30 12:45:17.895749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:17.895305 4413 scope.go:122] "RemoveContainer" containerID="67e3658fedf3ea8954df8a9ec6614e0113eed625af1ee24ade5f27609efaff89" Oct 30 12:45:17.895846 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:17.895828 4413 scope.go:122] "RemoveContainer" containerID="7aedc6a6daa5e5ef4db82de32b3e29c9f11bc5adba56a71b719a84f53edaa27f" Oct 30 12:45:17.896669 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:45:17.896365 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm_odh-ai-gateway-infra(614cb5f4-97b0-46f2-8d23-c50b5b673eab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" podUID="614cb5f4-97b0-46f2-8d23-c50b5b673eab" Oct 30 12:45:17.898463 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:17.898433 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" event={"ID":"e3b4a2cf-88e0-43af-a20d-f8d79f76368c","Type":"ContainerStarted","Data":"2e6b8a0c3416db08f0fef531d67b20abe91bcfeb9a9f37e7e80bf1d24d86cd22"} Oct 30 12:45:17.939698 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:17.939657 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm"] Oct 30 12:45:18.013814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:18.013780 4413 scope.go:122] "RemoveContainer" containerID="8c4a70ad0b4e12f3cbdb34782c0537cbddd5206871a31733a7b6dd4557e6d469" Oct 30 12:45:18.093562 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:18.093531 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" Oct 30 12:45:18.169762 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:18.169672 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d4b98ce-81d9-4aff-9c27-7c27119fc440-kube-api-access-xx2tq\" (UniqueName: \"kubernetes.io/projected/7d4b98ce-81d9-4aff-9c27-7c27119fc440-kube-api-access-xx2tq\") pod \"7d4b98ce-81d9-4aff-9c27-7c27119fc440\" (UID: \"7d4b98ce-81d9-4aff-9c27-7c27119fc440\") " Oct 30 12:45:18.169953 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:18.169763 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d4b98ce-81d9-4aff-9c27-7c27119fc440-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4b98ce-81d9-4aff-9c27-7c27119fc440-maas-api-service-ca\") pod \"7d4b98ce-81d9-4aff-9c27-7c27119fc440\" (UID: \"7d4b98ce-81d9-4aff-9c27-7c27119fc440\") " Oct 30 12:45:18.170647 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:18.170527 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4b98ce-81d9-4aff-9c27-7c27119fc440-maas-api-service-ca" pod "7d4b98ce-81d9-4aff-9c27-7c27119fc440" (UID: "7d4b98ce-81d9-4aff-9c27-7c27119fc440"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:18.172287 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:18.172224 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4b98ce-81d9-4aff-9c27-7c27119fc440-kube-api-access-xx2tq" pod "7d4b98ce-81d9-4aff-9c27-7c27119fc440" (UID: "7d4b98ce-81d9-4aff-9c27-7c27119fc440"). InnerVolumeSpecName "kube-api-access-xx2tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:18.272093 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:18.272057 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx2tq\" (UniqueName: \"kubernetes.io/projected/7d4b98ce-81d9-4aff-9c27-7c27119fc440-kube-api-access-xx2tq\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:18.272093 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:18.272087 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4b98ce-81d9-4aff-9c27-7c27119fc440-maas-api-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:18.911048 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:18.911021 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" Oct 30 12:45:18.911542 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:18.911062 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n" event={"ID":"7d4b98ce-81d9-4aff-9c27-7c27119fc440","Type":"ContainerDied","Data":"a223e58bfa21b0e853d6e2a9a3925265f65c6b61e4009614fe45abaeee7f7529"} Oct 30 12:45:18.911542 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:18.911134 4413 scope.go:122] "RemoveContainer" containerID="5c1c7225e899e19822b3dd4c49989a36ba5666005b1fb6c7d6923e83df8f5ffb" Oct 30 12:45:18.915982 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:18.915952 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" event={"ID":"9cb3d669-0630-425d-926f-fdfedac04d40","Type":"ContainerStarted","Data":"e1e5be0cb0875d66cd748682e89f9dbb694a0c730452d37da82e60414f94a6f0"} Oct 30 12:45:18.916476 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:18.916452 4413 scope.go:122] "RemoveContainer" containerID="e1e5be0cb0875d66cd748682e89f9dbb694a0c730452d37da82e60414f94a6f0" Oct 30 12:45:18.916901 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:45:18.916867 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7_odh-ai-gateway-infra(9cb3d669-0630-425d-926f-fdfedac04d40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" podUID="9cb3d669-0630-425d-926f-fdfedac04d40" Oct 30 12:45:18.918404 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:18.918382 4413 generic.go:356] "Generic (PLEG): container finished" podID="e3b4a2cf-88e0-43af-a20d-f8d79f76368c" containerID="2e6b8a0c3416db08f0fef531d67b20abe91bcfeb9a9f37e7e80bf1d24d86cd22" exitCode=60 Oct 30 12:45:18.918492 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:18.918446 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" event={"ID":"e3b4a2cf-88e0-43af-a20d-f8d79f76368c","Type":"ContainerDied","Data":"2e6b8a0c3416db08f0fef531d67b20abe91bcfeb9a9f37e7e80bf1d24d86cd22"} Oct 30 12:45:18.918814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:18.918799 4413 scope.go:122] "RemoveContainer" containerID="2e6b8a0c3416db08f0fef531d67b20abe91bcfeb9a9f37e7e80bf1d24d86cd22" Oct 30 12:45:18.919126 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:45:18.919109 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889405-jhtjj_odh-ai-gateway-infra(e3b4a2cf-88e0-43af-a20d-f8d79f76368c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" podUID="e3b4a2cf-88e0-43af-a20d-f8d79f76368c" Oct 30 12:45:18.930345 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:18.930235 4413 scope.go:122] "RemoveContainer" containerID="b51791aff9ad3a2908134b74cc7106f5ad3dffbe6420a1ca917fa866365f12c2" Oct 30 12:45:19.094722 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:19.094690 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n"] Oct 30 12:45:19.107536 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:19.107506 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" Oct 30 12:45:19.110091 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:19.110064 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-42220c-29889405-m5h7n"] Oct 30 12:45:19.128882 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:19.128852 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj"] Oct 30 12:45:19.186096 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:19.186014 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/614cb5f4-97b0-46f2-8d23-c50b5b673eab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/614cb5f4-97b0-46f2-8d23-c50b5b673eab-maas-api-service-ca\") pod \"614cb5f4-97b0-46f2-8d23-c50b5b673eab\" (UID: \"614cb5f4-97b0-46f2-8d23-c50b5b673eab\") " Oct 30 12:45:19.186266 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:19.186143 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/614cb5f4-97b0-46f2-8d23-c50b5b673eab-kube-api-access-gvt4d\" (UniqueName: \"kubernetes.io/projected/614cb5f4-97b0-46f2-8d23-c50b5b673eab-kube-api-access-gvt4d\") pod \"614cb5f4-97b0-46f2-8d23-c50b5b673eab\" (UID: \"614cb5f4-97b0-46f2-8d23-c50b5b673eab\") " Oct 30 12:45:19.186472 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:19.186445 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/614cb5f4-97b0-46f2-8d23-c50b5b673eab-maas-api-service-ca" pod "614cb5f4-97b0-46f2-8d23-c50b5b673eab" (UID: "614cb5f4-97b0-46f2-8d23-c50b5b673eab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:19.186703 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:19.186687 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/614cb5f4-97b0-46f2-8d23-c50b5b673eab-maas-api-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:19.188953 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:19.188927 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614cb5f4-97b0-46f2-8d23-c50b5b673eab-kube-api-access-gvt4d" pod "614cb5f4-97b0-46f2-8d23-c50b5b673eab" (UID: "614cb5f4-97b0-46f2-8d23-c50b5b673eab"). InnerVolumeSpecName "kube-api-access-gvt4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:19.220433 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:19.220401 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:19.288213 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:19.288182 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvt4d\" (UniqueName: \"kubernetes.io/projected/614cb5f4-97b0-46f2-8d23-c50b5b673eab-kube-api-access-gvt4d\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:19.929131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:19.929102 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" Oct 30 12:45:19.929131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:19.929121 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm" event={"ID":"614cb5f4-97b0-46f2-8d23-c50b5b673eab","Type":"ContainerDied","Data":"0a7d617acac6144213e920cf64933f4bf66b125af8be568026cd316e2976169e"} Oct 30 12:45:19.929720 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:19.929187 4413 scope.go:122] "RemoveContainer" containerID="7aedc6a6daa5e5ef4db82de32b3e29c9f11bc5adba56a71b719a84f53edaa27f" Oct 30 12:45:19.931685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:19.931658 4413 generic.go:356] "Generic (PLEG): container finished" podID="9cb3d669-0630-425d-926f-fdfedac04d40" containerID="e1e5be0cb0875d66cd748682e89f9dbb694a0c730452d37da82e60414f94a6f0" exitCode=60 Oct 30 12:45:19.931792 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:19.931733 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" event={"ID":"9cb3d669-0630-425d-926f-fdfedac04d40","Type":"ContainerDied","Data":"e1e5be0cb0875d66cd748682e89f9dbb694a0c730452d37da82e60414f94a6f0"} Oct 30 12:45:19.932290 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:19.932269 4413 scope.go:122] "RemoveContainer" containerID="e1e5be0cb0875d66cd748682e89f9dbb694a0c730452d37da82e60414f94a6f0" Oct 30 12:45:19.933119 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:45:19.932700 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7_odh-ai-gateway-infra(9cb3d669-0630-425d-926f-fdfedac04d40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" podUID="9cb3d669-0630-425d-926f-fdfedac04d40" Oct 30 12:45:19.941411 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:19.941393 4413 scope.go:122] "RemoveContainer" containerID="8c4a70ad0b4e12f3cbdb34782c0537cbddd5206871a31733a7b6dd4557e6d469" Oct 30 12:45:20.002753 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:20.002711 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7"] Oct 30 12:45:20.119817 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:20.119786 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" Oct 30 12:45:20.159944 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:20.159914 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm"] Oct 30 12:45:20.174872 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:20.174839 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a7148-29889405-qn7hm"] Oct 30 12:45:20.203420 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:20.203329 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b4a2cf-88e0-43af-a20d-f8d79f76368c-kube-api-access-jbwp4\" (UniqueName: \"kubernetes.io/projected/e3b4a2cf-88e0-43af-a20d-f8d79f76368c-kube-api-access-jbwp4\") pod \"e3b4a2cf-88e0-43af-a20d-f8d79f76368c\" (UID: \"e3b4a2cf-88e0-43af-a20d-f8d79f76368c\") " Oct 30 12:45:20.203420 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:20.203391 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3b4a2cf-88e0-43af-a20d-f8d79f76368c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b4a2cf-88e0-43af-a20d-f8d79f76368c-maas-api-service-ca\") pod \"e3b4a2cf-88e0-43af-a20d-f8d79f76368c\" (UID: \"e3b4a2cf-88e0-43af-a20d-f8d79f76368c\") " Oct 30 12:45:20.203810 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:20.203785 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b4a2cf-88e0-43af-a20d-f8d79f76368c-maas-api-service-ca" pod "e3b4a2cf-88e0-43af-a20d-f8d79f76368c" (UID: "e3b4a2cf-88e0-43af-a20d-f8d79f76368c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:20.203996 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:20.203980 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b4a2cf-88e0-43af-a20d-f8d79f76368c-maas-api-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:20.206114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:20.206091 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b4a2cf-88e0-43af-a20d-f8d79f76368c-kube-api-access-jbwp4" pod "e3b4a2cf-88e0-43af-a20d-f8d79f76368c" (UID: "e3b4a2cf-88e0-43af-a20d-f8d79f76368c"). InnerVolumeSpecName "kube-api-access-jbwp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:20.305909 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:20.305874 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbwp4\" (UniqueName: \"kubernetes.io/projected/e3b4a2cf-88e0-43af-a20d-f8d79f76368c-kube-api-access-jbwp4\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:20.945793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:20.945765 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" Oct 30 12:45:20.946215 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:20.945803 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj" event={"ID":"e3b4a2cf-88e0-43af-a20d-f8d79f76368c","Type":"ContainerDied","Data":"9a8ef2e8f01f82a8467b080065a812bc8a44dd4e2deb7a0c1659747025415ee0"} Oct 30 12:45:20.946215 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:20.945870 4413 scope.go:122] "RemoveContainer" containerID="2e6b8a0c3416db08f0fef531d67b20abe91bcfeb9a9f37e7e80bf1d24d86cd22" Oct 30 12:45:21.042618 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:21.042587 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="614cb5f4-97b0-46f2-8d23-c50b5b673eab" path="/var/lib/kubelet/pods/614cb5f4-97b0-46f2-8d23-c50b5b673eab/volumes" Oct 30 12:45:21.043450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:21.043422 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d4b98ce-81d9-4aff-9c27-7c27119fc440" path="/var/lib/kubelet/pods/7d4b98ce-81d9-4aff-9c27-7c27119fc440/volumes" Oct 30 12:45:21.043849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:21.043832 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj"] Oct 30 12:45:21.059744 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:21.059715 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889405-jhtjj"] Oct 30 12:45:21.135269 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:21.135226 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" Oct 30 12:45:21.220122 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:21.220026 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cb3d669-0630-425d-926f-fdfedac04d40-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb3d669-0630-425d-926f-fdfedac04d40-maas-api-service-ca\") pod \"9cb3d669-0630-425d-926f-fdfedac04d40\" (UID: \"9cb3d669-0630-425d-926f-fdfedac04d40\") " Oct 30 12:45:21.220328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:21.220206 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb3d669-0630-425d-926f-fdfedac04d40-kube-api-access-lc9mh\" (UniqueName: \"kubernetes.io/projected/9cb3d669-0630-425d-926f-fdfedac04d40-kube-api-access-lc9mh\") pod \"9cb3d669-0630-425d-926f-fdfedac04d40\" (UID: \"9cb3d669-0630-425d-926f-fdfedac04d40\") " Oct 30 12:45:21.220452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:21.220428 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb3d669-0630-425d-926f-fdfedac04d40-maas-api-service-ca" pod "9cb3d669-0630-425d-926f-fdfedac04d40" (UID: "9cb3d669-0630-425d-926f-fdfedac04d40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:21.220892 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:21.220872 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cb3d669-0630-425d-926f-fdfedac04d40-maas-api-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:21.222803 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:21.222779 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb3d669-0630-425d-926f-fdfedac04d40-kube-api-access-lc9mh" pod "9cb3d669-0630-425d-926f-fdfedac04d40" (UID: "9cb3d669-0630-425d-926f-fdfedac04d40"). InnerVolumeSpecName "kube-api-access-lc9mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:21.322642 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:21.322603 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc9mh\" (UniqueName: \"kubernetes.io/projected/9cb3d669-0630-425d-926f-fdfedac04d40-kube-api-access-lc9mh\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:21.524628 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:21.524549 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:21.958190 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:21.958148 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" Oct 30 12:45:21.958190 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:21.958170 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7" event={"ID":"9cb3d669-0630-425d-926f-fdfedac04d40","Type":"ContainerDied","Data":"2850a3a169d1413f9ad9b1c37f522fa9b4c1f785c44fa97b6f573ec6a50d6bf2"} Oct 30 12:45:21.958822 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:21.958287 4413 scope.go:122] "RemoveContainer" containerID="e1e5be0cb0875d66cd748682e89f9dbb694a0c730452d37da82e60414f94a6f0" Oct 30 12:45:22.014303 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:22.014267 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7"] Oct 30 12:45:22.025470 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:22.025440 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12fd33-29889405-wmqq7"] Oct 30 12:45:23.032516 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:23.032480 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb3d669-0630-425d-926f-fdfedac04d40" path="/var/lib/kubelet/pods/9cb3d669-0630-425d-926f-fdfedac04d40/volumes" Oct 30 12:45:23.032979 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:23.032964 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3b4a2cf-88e0-43af-a20d-f8d79f76368c" path="/var/lib/kubelet/pods/e3b4a2cf-88e0-43af-a20d-f8d79f76368c/volumes" Oct 30 12:45:23.459846 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:23.459803 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:25.305823 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:25.304809 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:27.065162 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:27.065125 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:27.105809 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:27.105773 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985"] Oct 30 12:45:27.106048 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:27.106010 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" podUID="b3cbdc0e-806c-47f0-81dc-f09209554793" containerName="istio-proxy" containerID="cri-o://56b75b93d39bc4fb81f296b4e3be09fbdc0db125411a64f285775c5a7ba536fd" gracePeriod=30 Oct 30 12:45:28.891878 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:28.891846 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:29.249018 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:29.248927 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:31.224770 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:31.224734 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:31.482932 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:31.482841 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:32.408762 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.408732 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:45:32.498909 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.498876 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3cbdc0e-806c-47f0-81dc-f09209554793-kube-api-access-wlngn\" (UniqueName: \"kubernetes.io/projected/b3cbdc0e-806c-47f0-81dc-f09209554793-kube-api-access-wlngn\") pod \"b3cbdc0e-806c-47f0-81dc-f09209554793\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " Oct 30 12:45:32.499106 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.498992 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-workload-certs\") pod \"b3cbdc0e-806c-47f0-81dc-f09209554793\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " Oct 30 12:45:32.499177 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.499131 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-workload-socket\") pod \"b3cbdc0e-806c-47f0-81dc-f09209554793\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " Oct 30 12:45:32.499231 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.499203 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-data\") pod \"b3cbdc0e-806c-47f0-81dc-f09209554793\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " Oct 30 12:45:32.499353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.499339 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3cbdc0e-806c-47f0-81dc-f09209554793-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cbdc0e-806c-47f0-81dc-f09209554793-istiod-ca-cert\") pod \"b3cbdc0e-806c-47f0-81dc-f09209554793\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " Oct 30 12:45:32.499415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.499396 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-credential-socket\") pod \"b3cbdc0e-806c-47f0-81dc-f09209554793\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " Oct 30 12:45:32.499483 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.499399 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-workload-certs" pod "b3cbdc0e-806c-47f0-81dc-f09209554793" (UID: "b3cbdc0e-806c-47f0-81dc-f09209554793"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:32.499483 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.499440 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-token\" (UniqueName: \"kubernetes.io/projected/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-token\") pod \"b3cbdc0e-806c-47f0-81dc-f09209554793\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " Oct 30 12:45:32.499586 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.499487 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-envoy\") pod \"b3cbdc0e-806c-47f0-81dc-f09209554793\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " Oct 30 12:45:32.499586 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.499550 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-podinfo\") pod \"b3cbdc0e-806c-47f0-81dc-f09209554793\" (UID: \"b3cbdc0e-806c-47f0-81dc-f09209554793\") " Oct 30 12:45:32.499681 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.499643 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-credential-socket" pod "b3cbdc0e-806c-47f0-81dc-f09209554793" (UID: "b3cbdc0e-806c-47f0-81dc-f09209554793"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:32.499768 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.499736 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3cbdc0e-806c-47f0-81dc-f09209554793-istiod-ca-cert" pod "b3cbdc0e-806c-47f0-81dc-f09209554793" (UID: "b3cbdc0e-806c-47f0-81dc-f09209554793"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:32.500053 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.499881 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-data" pod "b3cbdc0e-806c-47f0-81dc-f09209554793" (UID: "b3cbdc0e-806c-47f0-81dc-f09209554793"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:32.500540 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.500516 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.500851 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.500838 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.501001 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.500988 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cbdc0e-806c-47f0-81dc-f09209554793-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.501123 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.501110 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.501190 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.500567 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-workload-socket" pod "b3cbdc0e-806c-47f0-81dc-f09209554793" (UID: "b3cbdc0e-806c-47f0-81dc-f09209554793"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:32.502616 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.502580 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-podinfo" pod "b3cbdc0e-806c-47f0-81dc-f09209554793" (UID: "b3cbdc0e-806c-47f0-81dc-f09209554793"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:45:32.502717 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.502647 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-token" pod "b3cbdc0e-806c-47f0-81dc-f09209554793" (UID: "b3cbdc0e-806c-47f0-81dc-f09209554793"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:32.502828 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.502802 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cbdc0e-806c-47f0-81dc-f09209554793-kube-api-access-wlngn" pod "b3cbdc0e-806c-47f0-81dc-f09209554793" (UID: "b3cbdc0e-806c-47f0-81dc-f09209554793"). InnerVolumeSpecName "kube-api-access-wlngn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:32.503068 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.503043 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-envoy" pod "b3cbdc0e-806c-47f0-81dc-f09209554793" (UID: "b3cbdc0e-806c-47f0-81dc-f09209554793"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:45:32.602918 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.602884 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.602918 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.602913 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.602918 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.602923 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.602918 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.602930 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cbdc0e-806c-47f0-81dc-f09209554793-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:32.603201 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:32.602939 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlngn\" (UniqueName: \"kubernetes.io/projected/b3cbdc0e-806c-47f0-81dc-f09209554793-kube-api-access-wlngn\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:33.050904 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:33.050867 4413 generic.go:356] "Generic (PLEG): container finished" podID="b3cbdc0e-806c-47f0-81dc-f09209554793" containerID="56b75b93d39bc4fb81f296b4e3be09fbdc0db125411a64f285775c5a7ba536fd" exitCode=0 Oct 30 12:45:33.051107 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:33.050921 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" Oct 30 12:45:33.051107 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:33.050949 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" event={"ID":"b3cbdc0e-806c-47f0-81dc-f09209554793","Type":"ContainerDied","Data":"56b75b93d39bc4fb81f296b4e3be09fbdc0db125411a64f285775c5a7ba536fd"} Oct 30 12:45:33.051107 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:33.051004 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985" event={"ID":"b3cbdc0e-806c-47f0-81dc-f09209554793","Type":"ContainerDied","Data":"84321ad12c6f8f65a6cec41437c351868441aca8fe81d8fa8f1ae4b87abdaf32"} Oct 30 12:45:33.051107 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:33.051025 4413 scope.go:122] "RemoveContainer" containerID="56b75b93d39bc4fb81f296b4e3be09fbdc0db125411a64f285775c5a7ba536fd" Oct 30 12:45:33.061872 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:33.061851 4413 scope.go:122] "RemoveContainer" containerID="56b75b93d39bc4fb81f296b4e3be09fbdc0db125411a64f285775c5a7ba536fd" Oct 30 12:45:33.062196 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:45:33.062170 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b75b93d39bc4fb81f296b4e3be09fbdc0db125411a64f285775c5a7ba536fd\": container with ID starting with 56b75b93d39bc4fb81f296b4e3be09fbdc0db125411a64f285775c5a7ba536fd not found: ID does not exist" containerID="56b75b93d39bc4fb81f296b4e3be09fbdc0db125411a64f285775c5a7ba536fd" Oct 30 12:45:33.062295 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:33.062203 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b75b93d39bc4fb81f296b4e3be09fbdc0db125411a64f285775c5a7ba536fd"} err="failed to get container status \"56b75b93d39bc4fb81f296b4e3be09fbdc0db125411a64f285775c5a7ba536fd\": rpc error: code = NotFound desc = could not find container \"56b75b93d39bc4fb81f296b4e3be09fbdc0db125411a64f285775c5a7ba536fd\": container with ID starting with 56b75b93d39bc4fb81f296b4e3be09fbdc0db125411a64f285775c5a7ba536fd not found: ID does not exist" Oct 30 12:45:33.125099 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:33.125065 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985"] Oct 30 12:45:33.136810 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:33.136774 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-z4985"] Oct 30 12:45:33.375110 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:33.375080 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:34.017034 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.017002 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:34.856375 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.856340 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l"] Oct 30 12:45:34.857170 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857153 4413 cpu_manager.go:414] "containerMap: removing container" podUID="9cb3d669-0630-425d-926f-fdfedac04d40" containerName="cleanup" Oct 30 12:45:34.857264 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857172 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb3d669-0630-425d-926f-fdfedac04d40" containerName="cleanup" Oct 30 12:45:34.857264 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857184 4413 cpu_manager.go:414] "containerMap: removing container" podUID="7d4b98ce-81d9-4aff-9c27-7c27119fc440" containerName="cleanup" Oct 30 12:45:34.857264 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857191 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d4b98ce-81d9-4aff-9c27-7c27119fc440" containerName="cleanup" Oct 30 12:45:34.857264 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857217 4413 cpu_manager.go:414] "containerMap: removing container" podUID="7d4b98ce-81d9-4aff-9c27-7c27119fc440" containerName="cleanup" Oct 30 12:45:34.857264 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857222 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d4b98ce-81d9-4aff-9c27-7c27119fc440" containerName="cleanup" Oct 30 12:45:34.857264 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857232 4413 cpu_manager.go:414] "containerMap: removing container" podUID="614cb5f4-97b0-46f2-8d23-c50b5b673eab" containerName="cleanup" Oct 30 12:45:34.857264 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857237 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="614cb5f4-97b0-46f2-8d23-c50b5b673eab" containerName="cleanup" Oct 30 12:45:34.857264 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857269 4413 cpu_manager.go:414] "containerMap: removing container" podUID="9cb3d669-0630-425d-926f-fdfedac04d40" containerName="cleanup" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857274 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb3d669-0630-425d-926f-fdfedac04d40" containerName="cleanup" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857282 4413 cpu_manager.go:414] "containerMap: removing container" podUID="614cb5f4-97b0-46f2-8d23-c50b5b673eab" containerName="cleanup" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857286 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="614cb5f4-97b0-46f2-8d23-c50b5b673eab" containerName="cleanup" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857303 4413 cpu_manager.go:414] "containerMap: removing container" podUID="e3b4a2cf-88e0-43af-a20d-f8d79f76368c" containerName="cleanup" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857307 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b4a2cf-88e0-43af-a20d-f8d79f76368c" containerName="cleanup" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857313 4413 cpu_manager.go:414] "containerMap: removing container" podUID="b3cbdc0e-806c-47f0-81dc-f09209554793" containerName="istio-proxy" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857318 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3cbdc0e-806c-47f0-81dc-f09209554793" containerName="istio-proxy" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857339 4413 cpu_manager.go:414] "containerMap: removing container" podUID="e3b4a2cf-88e0-43af-a20d-f8d79f76368c" containerName="cleanup" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857343 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b4a2cf-88e0-43af-a20d-f8d79f76368c" containerName="cleanup" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857467 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb3d669-0630-425d-926f-fdfedac04d40" containerName="cleanup" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857475 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="614cb5f4-97b0-46f2-8d23-c50b5b673eab" containerName="cleanup" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857483 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b4a2cf-88e0-43af-a20d-f8d79f76368c" containerName="cleanup" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857489 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3cbdc0e-806c-47f0-81dc-f09209554793" containerName="istio-proxy" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857502 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d4b98ce-81d9-4aff-9c27-7c27119fc440" containerName="cleanup" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857509 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b4a2cf-88e0-43af-a20d-f8d79f76368c" containerName="cleanup" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857515 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb3d669-0630-425d-926f-fdfedac04d40" containerName="cleanup" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857522 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d4b98ce-81d9-4aff-9c27-7c27119fc440" containerName="cleanup" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857528 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d4b98ce-81d9-4aff-9c27-7c27119fc440" containerName="cleanup" Oct 30 12:45:34.857716 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.857535 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="614cb5f4-97b0-46f2-8d23-c50b5b673eab" containerName="cleanup" Oct 30 12:45:34.863167 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.863140 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" Oct 30 12:45:34.880853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.880821 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-10f364-serving-cert\"" Oct 30 12:45:34.891130 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.891101 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l"] Oct 30 12:45:34.995692 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:34.995654 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:35.034189 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:35.034150 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3cbdc0e-806c-47f0-81dc-f09209554793" path="/var/lib/kubelet/pods/b3cbdc0e-806c-47f0-81dc-f09209554793/volumes" Oct 30 12:45:35.039922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:35.039832 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l\" (UID: \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" Oct 30 12:45:35.039922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:35.039883 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-metrics-certs\") pod \"maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l\" (UID: \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" Oct 30 12:45:35.040354 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:35.040177 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l\" (UID: \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" Oct 30 12:45:35.040354 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:35.040226 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6p7d\" (UniqueName: \"kubernetes.io/projected/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-kube-api-access-d6p7d\") pod \"maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l\" (UID: \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" Oct 30 12:45:35.141294 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:35.141234 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l\" (UID: \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" Oct 30 12:45:35.141518 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:35.141317 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6p7d\" (UniqueName: \"kubernetes.io/projected/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-kube-api-access-d6p7d\") pod \"maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l\" (UID: \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" Oct 30 12:45:35.141518 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:35.141375 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l\" (UID: \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" Oct 30 12:45:35.141518 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:35.141395 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-metrics-certs\") pod \"maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l\" (UID: \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" Oct 30 12:45:35.142005 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:35.141977 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l\" (UID: \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" Oct 30 12:45:35.144018 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:35.143995 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-metrics-certs\") pod \"maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l\" (UID: \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" Oct 30 12:45:35.144158 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:35.144048 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l\" (UID: \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" Oct 30 12:45:35.145553 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:35.145529 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:35.169029 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:35.168992 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6p7d\" (UniqueName: \"kubernetes.io/projected/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-kube-api-access-d6p7d\") pod \"maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l\" (UID: \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" Oct 30 12:45:35.185852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:35.185802 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" Oct 30 12:45:35.329207 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:35.329176 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:35.508990 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:35.508961 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l"] Oct 30 12:45:35.509360 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:45:35.509337 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa9879d_dab4_42ea_8dd8_ad6bba64214c.slice/crio-5e7c7715eb4809401562e37158d69ba0ff878a6ff768073e001d2ba144d045c2 WatchSource:0}: Error finding container 5e7c7715eb4809401562e37158d69ba0ff878a6ff768073e001d2ba144d045c2: Status 404 returned error can't find the container with id 5e7c7715eb4809401562e37158d69ba0ff878a6ff768073e001d2ba144d045c2 Oct 30 12:45:35.816145 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:45:35.816049 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:45:36.131149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:36.131102 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" event={"ID":"4aa9879d-dab4-42ea-8dd8-ad6bba64214c","Type":"ContainerStarted","Data":"5e7c7715eb4809401562e37158d69ba0ff878a6ff768073e001d2ba144d045c2"} Oct 30 12:45:36.985887 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:36.985844 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:37.158272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:37.158136 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" event={"ID":"4aa9879d-dab4-42ea-8dd8-ad6bba64214c","Type":"ContainerStarted","Data":"7c7d28c6baf25f3824751a21c640b20082a55a88b1aa783269d24a70f16deaff"} Oct 30 12:45:37.158272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:37.158230 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" Oct 30 12:45:37.204483 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:37.204424 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" podStartSLOduration=2.705625863 podStartE2EDuration="3.20440807s" totalImagesPullingTime="498.782207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:37.203316423 +0000 UTC m=+3090.313159305" watchObservedRunningTime="2026-10-30 12:45:37.20440807 +0000 UTC m=+3090.314250950" Oct 30 12:45:37.476764 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:37.476734 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:38.853399 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:38.853358 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:39.929291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:39.929241 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:40.128463 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:40.128431 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:40.554672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:40.554639 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:40.753596 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:40.753563 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:41.142523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:41.142489 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:41.284705 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:41.284657 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:41.601468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:41.601390 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2"] Oct 30 12:45:41.703295 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:41.703227 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:41.855801 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:41.855712 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:42.605700 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:42.605667 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:43.172016 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:43.171990 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" Oct 30 12:45:44.354891 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:44.354862 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:44.423613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:44.423579 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576"] Oct 30 12:45:46.053122 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:46.053085 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:46.602464 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:46.602425 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" podUID="5ad807e4-0d66-4856-964a-5b7a4a3a80bb" containerName="payload-processing" containerID="cri-o://1e9ded4470f50828b2d0be864d90d08c1b8432d986256111a7f944b0082fb401" gracePeriod=25 Oct 30 12:45:46.877577 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:46.877550 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" Oct 30 12:45:46.885970 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:46.885944 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:46.915497 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:46.915464 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-routing-overlay-volume\") pod \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\" (UID: \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\") " Oct 30 12:45:46.915668 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:46.915550 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-kube-api-access-tt4cq\" (UniqueName: \"kubernetes.io/projected/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-kube-api-access-tt4cq\") pod \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\" (UID: \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\") " Oct 30 12:45:46.915668 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:46.915647 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-provider-credentials\") pod \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\" (UID: \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\") " Oct 30 12:45:46.915759 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:46.915721 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-plugins-config-volume\") pod \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\" (UID: \"5ad807e4-0d66-4856-964a-5b7a4a3a80bb\") " Oct 30 12:45:46.915921 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:46.915885 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-routing-overlay-volume" pod "5ad807e4-0d66-4856-964a-5b7a4a3a80bb" (UID: "5ad807e4-0d66-4856-964a-5b7a4a3a80bb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:46.916216 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:46.916153 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-plugins-config-volume" pod "5ad807e4-0d66-4856-964a-5b7a4a3a80bb" (UID: "5ad807e4-0d66-4856-964a-5b7a4a3a80bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:46.916448 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:46.916332 4413 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-routing-overlay-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:46.916448 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:46.916349 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:46.931544 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:46.931487 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-kube-api-access-tt4cq" pod "5ad807e4-0d66-4856-964a-5b7a4a3a80bb" (UID: "5ad807e4-0d66-4856-964a-5b7a4a3a80bb"). InnerVolumeSpecName "kube-api-access-tt4cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:46.935864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:46.935810 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-provider-credentials" pod "5ad807e4-0d66-4856-964a-5b7a4a3a80bb" (UID: "5ad807e4-0d66-4856-964a-5b7a4a3a80bb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:47.017459 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:47.017428 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:47.017673 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:47.017561 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt4cq\" (UniqueName: \"kubernetes.io/projected/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-kube-api-access-tt4cq\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:47.017673 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:47.017585 4413 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ad807e4-0d66-4856-964a-5b7a4a3a80bb-provider-credentials\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:47.203814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:47.203784 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:47.279030 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:47.278999 4413 generic.go:356] "Generic (PLEG): container finished" podID="5ad807e4-0d66-4856-964a-5b7a4a3a80bb" containerID="1e9ded4470f50828b2d0be864d90d08c1b8432d986256111a7f944b0082fb401" exitCode=0 Oct 30 12:45:47.279206 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:47.279061 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" Oct 30 12:45:47.279206 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:47.279103 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" event={"ID":"5ad807e4-0d66-4856-964a-5b7a4a3a80bb","Type":"ContainerDied","Data":"1e9ded4470f50828b2d0be864d90d08c1b8432d986256111a7f944b0082fb401"} Oct 30 12:45:47.279206 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:47.279168 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2" event={"ID":"5ad807e4-0d66-4856-964a-5b7a4a3a80bb","Type":"ContainerDied","Data":"aa538f650a79d0f63d5f03eb12219f9fcaa128280b08ef6f9ae865173b2bf140"} Oct 30 12:45:47.279206 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:47.279198 4413 scope.go:122] "RemoveContainer" containerID="1e9ded4470f50828b2d0be864d90d08c1b8432d986256111a7f944b0082fb401" Oct 30 12:45:47.290353 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:47.290332 4413 scope.go:122] "RemoveContainer" containerID="1e9ded4470f50828b2d0be864d90d08c1b8432d986256111a7f944b0082fb401" Oct 30 12:45:47.290626 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:45:47.290608 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e9ded4470f50828b2d0be864d90d08c1b8432d986256111a7f944b0082fb401\": container with ID starting with 1e9ded4470f50828b2d0be864d90d08c1b8432d986256111a7f944b0082fb401 not found: ID does not exist" containerID="1e9ded4470f50828b2d0be864d90d08c1b8432d986256111a7f944b0082fb401" Oct 30 12:45:47.290678 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:47.290637 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e9ded4470f50828b2d0be864d90d08c1b8432d986256111a7f944b0082fb401"} err="failed to get container status \"1e9ded4470f50828b2d0be864d90d08c1b8432d986256111a7f944b0082fb401\": rpc error: code = NotFound desc = could not find container \"1e9ded4470f50828b2d0be864d90d08c1b8432d986256111a7f944b0082fb401\": container with ID starting with 1e9ded4470f50828b2d0be864d90d08c1b8432d986256111a7f944b0082fb401 not found: ID does not exist" Oct 30 12:45:47.353824 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:47.353790 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2"] Oct 30 12:45:47.365561 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:47.365532 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0f7d14/payload-processing-external-model-e2e-extmod-b-0f7d14-8799qvws2"] Oct 30 12:45:47.761793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:47.761762 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:48.558683 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:48.558650 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:48.710550 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:48.710518 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:48.849118 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:48.849036 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:49.023518 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:49.023446 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad807e4-0d66-4856-964a-5b7a4a3a80bb" path="/var/lib/kubelet/pods/5ad807e4-0d66-4856-964a-5b7a4a3a80bb/volumes" Oct 30 12:45:49.424504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:49.424448 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" podUID="4cd33196-cd20-4832-9cb0-53063fb91c11" containerName="payload-processing" containerID="cri-o://ce3c53632f047f36e60ff49ff03d729b2746367aaa3fababde2ead60f268e7d9" gracePeriod=25 Oct 30 12:45:49.696392 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:49.696364 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:49.730561 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:49.730532 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" Oct 30 12:45:49.860383 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:49.860355 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd33196-cd20-4832-9cb0-53063fb91c11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd33196-cd20-4832-9cb0-53063fb91c11-plugins-config-volume\") pod \"4cd33196-cd20-4832-9cb0-53063fb91c11\" (UID: \"4cd33196-cd20-4832-9cb0-53063fb91c11\") " Oct 30 12:45:49.860564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:49.860406 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd33196-cd20-4832-9cb0-53063fb91c11-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4cd33196-cd20-4832-9cb0-53063fb91c11-provider-credentials\") pod \"4cd33196-cd20-4832-9cb0-53063fb91c11\" (UID: \"4cd33196-cd20-4832-9cb0-53063fb91c11\") " Oct 30 12:45:49.860607 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:49.860597 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd33196-cd20-4832-9cb0-53063fb91c11-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4cd33196-cd20-4832-9cb0-53063fb91c11-routing-overlay-volume\") pod \"4cd33196-cd20-4832-9cb0-53063fb91c11\" (UID: \"4cd33196-cd20-4832-9cb0-53063fb91c11\") " Oct 30 12:45:49.860679 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:49.860667 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd33196-cd20-4832-9cb0-53063fb91c11-kube-api-access-csx45\" (UniqueName: \"kubernetes.io/projected/4cd33196-cd20-4832-9cb0-53063fb91c11-kube-api-access-csx45\") pod \"4cd33196-cd20-4832-9cb0-53063fb91c11\" (UID: \"4cd33196-cd20-4832-9cb0-53063fb91c11\") " Oct 30 12:45:49.860894 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:49.860863 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd33196-cd20-4832-9cb0-53063fb91c11-plugins-config-volume" pod "4cd33196-cd20-4832-9cb0-53063fb91c11" (UID: "4cd33196-cd20-4832-9cb0-53063fb91c11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:49.861041 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:49.860968 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd33196-cd20-4832-9cb0-53063fb91c11-routing-overlay-volume" pod "4cd33196-cd20-4832-9cb0-53063fb91c11" (UID: "4cd33196-cd20-4832-9cb0-53063fb91c11"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:45:49.861235 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:49.861217 4413 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4cd33196-cd20-4832-9cb0-53063fb91c11-routing-overlay-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:49.861322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:49.861238 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd33196-cd20-4832-9cb0-53063fb91c11-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:49.862985 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:49.862967 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd33196-cd20-4832-9cb0-53063fb91c11-kube-api-access-csx45" pod "4cd33196-cd20-4832-9cb0-53063fb91c11" (UID: "4cd33196-cd20-4832-9cb0-53063fb91c11"). InnerVolumeSpecName "kube-api-access-csx45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:49.863047 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:49.862994 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd33196-cd20-4832-9cb0-53063fb91c11-provider-credentials" pod "4cd33196-cd20-4832-9cb0-53063fb91c11" (UID: "4cd33196-cd20-4832-9cb0-53063fb91c11"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:45:49.963273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:49.963160 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csx45\" (UniqueName: \"kubernetes.io/projected/4cd33196-cd20-4832-9cb0-53063fb91c11-kube-api-access-csx45\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:49.963273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:49.963190 4413 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4cd33196-cd20-4832-9cb0-53063fb91c11-provider-credentials\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:45:50.303630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:50.303542 4413 generic.go:356] "Generic (PLEG): container finished" podID="4cd33196-cd20-4832-9cb0-53063fb91c11" containerID="ce3c53632f047f36e60ff49ff03d729b2746367aaa3fababde2ead60f268e7d9" exitCode=0 Oct 30 12:45:50.303630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:50.303586 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" event={"ID":"4cd33196-cd20-4832-9cb0-53063fb91c11","Type":"ContainerDied","Data":"ce3c53632f047f36e60ff49ff03d729b2746367aaa3fababde2ead60f268e7d9"} Oct 30 12:45:50.303630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:50.303590 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" Oct 30 12:45:50.303630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:50.303625 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576" event={"ID":"4cd33196-cd20-4832-9cb0-53063fb91c11","Type":"ContainerDied","Data":"7cc9e55b0cdf0144deb50abad9b974b585078253af03bf69cf30107c4788bc4a"} Oct 30 12:45:50.303985 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:50.303650 4413 scope.go:122] "RemoveContainer" containerID="ce3c53632f047f36e60ff49ff03d729b2746367aaa3fababde2ead60f268e7d9" Oct 30 12:45:50.320504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:50.320487 4413 scope.go:122] "RemoveContainer" containerID="ce3c53632f047f36e60ff49ff03d729b2746367aaa3fababde2ead60f268e7d9" Oct 30 12:45:50.320792 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:45:50.320766 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce3c53632f047f36e60ff49ff03d729b2746367aaa3fababde2ead60f268e7d9\": container with ID starting with ce3c53632f047f36e60ff49ff03d729b2746367aaa3fababde2ead60f268e7d9 not found: ID does not exist" containerID="ce3c53632f047f36e60ff49ff03d729b2746367aaa3fababde2ead60f268e7d9" Oct 30 12:45:50.320838 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:50.320802 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce3c53632f047f36e60ff49ff03d729b2746367aaa3fababde2ead60f268e7d9"} err="failed to get container status \"ce3c53632f047f36e60ff49ff03d729b2746367aaa3fababde2ead60f268e7d9\": rpc error: code = NotFound desc = could not find container \"ce3c53632f047f36e60ff49ff03d729b2746367aaa3fababde2ead60f268e7d9\": container with ID starting with ce3c53632f047f36e60ff49ff03d729b2746367aaa3fababde2ead60f268e7d9 not found: ID does not exist" Oct 30 12:45:50.358685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:50.358644 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576"] Oct 30 12:45:50.369486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:50.369454 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6km576"] Oct 30 12:45:51.021620 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:51.021577 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd33196-cd20-4832-9cb0-53063fb91c11" path="/var/lib/kubelet/pods/4cd33196-cd20-4832-9cb0-53063fb91c11/volumes" Oct 30 12:45:51.401794 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:51.401757 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:52.294110 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:52.294072 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:52.431767 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:52.431726 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:53.158580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:53.158545 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:53.285818 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:53.285782 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:54.418050 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:54.418012 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:54.570561 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:54.570537 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:54.970492 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:54.970457 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:55.185946 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:55.185911 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:55.395803 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:55.395773 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:56.392955 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:56.392919 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:57.265700 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:57.265666 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:57.427336 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:57.427307 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:57.561522 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:57.561445 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:57.831043 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:57.830973 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:58.088220 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:58.088145 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:58.630261 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:58.630214 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:45:59.384424 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:45:59.384395 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:00.204994 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:00.204962 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:00.918843 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:00.918806 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-7lb7d" podUID="1a3afa91-aff6-455b-aeda-7275cd8ac8e0" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:46:00.919020 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:00.918919 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7lb7d" Oct 30 12:46:00.919561 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:00.919536 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-7lb7d" podUID="1a3afa91-aff6-455b-aeda-7275cd8ac8e0" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:46:01.087239 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:01.087206 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:01.329029 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:01.328945 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:02.977013 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:02.976983 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:04.572114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:04.572082 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:04.972942 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:04.972908 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr"] Oct 30 12:46:04.989703 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:04.989664 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:05.113010 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:05.112966 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:05.645200 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:05.645163 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:05.826367 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:46:05.826307 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:46:06.269743 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:06.269710 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:07.854478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:07.854443 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:09.480971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:09.480939 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:09.974304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:09.974222 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" podUID="7fa60d02-3a82-4f79-9382-bc3a84180c08" containerName="payload-processing" containerID="cri-o://a2b1027cc6cd3d31f9fba667369555c70f17053c99c6d5cc548a4be79a579760" gracePeriod=25 Oct 30 12:46:10.276498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.276461 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" Oct 30 12:46:10.408734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.408703 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa60d02-3a82-4f79-9382-bc3a84180c08-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fa60d02-3a82-4f79-9382-bc3a84180c08-routing-overlay-volume\") pod \"7fa60d02-3a82-4f79-9382-bc3a84180c08\" (UID: \"7fa60d02-3a82-4f79-9382-bc3a84180c08\") " Oct 30 12:46:10.408940 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.408762 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa60d02-3a82-4f79-9382-bc3a84180c08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa60d02-3a82-4f79-9382-bc3a84180c08-plugins-config-volume\") pod \"7fa60d02-3a82-4f79-9382-bc3a84180c08\" (UID: \"7fa60d02-3a82-4f79-9382-bc3a84180c08\") " Oct 30 12:46:10.409001 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.408970 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa60d02-3a82-4f79-9382-bc3a84180c08-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fa60d02-3a82-4f79-9382-bc3a84180c08-provider-credentials\") pod \"7fa60d02-3a82-4f79-9382-bc3a84180c08\" (UID: \"7fa60d02-3a82-4f79-9382-bc3a84180c08\") " Oct 30 12:46:10.409052 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.409044 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa60d02-3a82-4f79-9382-bc3a84180c08-kube-api-access-qsm6t\" (UniqueName: \"kubernetes.io/projected/7fa60d02-3a82-4f79-9382-bc3a84180c08-kube-api-access-qsm6t\") pod \"7fa60d02-3a82-4f79-9382-bc3a84180c08\" (UID: \"7fa60d02-3a82-4f79-9382-bc3a84180c08\") " Oct 30 12:46:10.409188 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.409158 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa60d02-3a82-4f79-9382-bc3a84180c08-routing-overlay-volume" pod "7fa60d02-3a82-4f79-9382-bc3a84180c08" (UID: "7fa60d02-3a82-4f79-9382-bc3a84180c08"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:46:10.409284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.409201 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa60d02-3a82-4f79-9382-bc3a84180c08-plugins-config-volume" pod "7fa60d02-3a82-4f79-9382-bc3a84180c08" (UID: "7fa60d02-3a82-4f79-9382-bc3a84180c08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:46:10.409694 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.409669 4413 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7fa60d02-3a82-4f79-9382-bc3a84180c08-routing-overlay-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:46:10.409694 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.409695 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fa60d02-3a82-4f79-9382-bc3a84180c08-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:46:10.411516 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.411493 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa60d02-3a82-4f79-9382-bc3a84180c08-provider-credentials" pod "7fa60d02-3a82-4f79-9382-bc3a84180c08" (UID: "7fa60d02-3a82-4f79-9382-bc3a84180c08"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:10.411643 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.411538 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa60d02-3a82-4f79-9382-bc3a84180c08-kube-api-access-qsm6t" pod "7fa60d02-3a82-4f79-9382-bc3a84180c08" (UID: "7fa60d02-3a82-4f79-9382-bc3a84180c08"). InnerVolumeSpecName "kube-api-access-qsm6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:46:10.508539 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.508449 4413 generic.go:356] "Generic (PLEG): container finished" podID="7fa60d02-3a82-4f79-9382-bc3a84180c08" containerID="a2b1027cc6cd3d31f9fba667369555c70f17053c99c6d5cc548a4be79a579760" exitCode=0 Oct 30 12:46:10.508539 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.508500 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" Oct 30 12:46:10.509025 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.508541 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" event={"ID":"7fa60d02-3a82-4f79-9382-bc3a84180c08","Type":"ContainerDied","Data":"a2b1027cc6cd3d31f9fba667369555c70f17053c99c6d5cc548a4be79a579760"} Oct 30 12:46:10.509025 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.508604 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr" event={"ID":"7fa60d02-3a82-4f79-9382-bc3a84180c08","Type":"ContainerDied","Data":"ff3bc7332df0db5e341bdb5ef3ed3eb839080b8e5aadf9054877335d1c393f25"} Oct 30 12:46:10.509025 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.508634 4413 scope.go:122] "RemoveContainer" containerID="a2b1027cc6cd3d31f9fba667369555c70f17053c99c6d5cc548a4be79a579760" Oct 30 12:46:10.511118 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.511097 4413 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7fa60d02-3a82-4f79-9382-bc3a84180c08-provider-credentials\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:46:10.511218 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.511122 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsm6t\" (UniqueName: \"kubernetes.io/projected/7fa60d02-3a82-4f79-9382-bc3a84180c08-kube-api-access-qsm6t\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:46:10.520457 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.520439 4413 scope.go:122] "RemoveContainer" containerID="a2b1027cc6cd3d31f9fba667369555c70f17053c99c6d5cc548a4be79a579760" Oct 30 12:46:10.520751 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:46:10.520732 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b1027cc6cd3d31f9fba667369555c70f17053c99c6d5cc548a4be79a579760\": container with ID starting with a2b1027cc6cd3d31f9fba667369555c70f17053c99c6d5cc548a4be79a579760 not found: ID does not exist" containerID="a2b1027cc6cd3d31f9fba667369555c70f17053c99c6d5cc548a4be79a579760" Oct 30 12:46:10.520799 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.520760 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b1027cc6cd3d31f9fba667369555c70f17053c99c6d5cc548a4be79a579760"} err="failed to get container status \"a2b1027cc6cd3d31f9fba667369555c70f17053c99c6d5cc548a4be79a579760\": rpc error: code = NotFound desc = could not find container \"a2b1027cc6cd3d31f9fba667369555c70f17053c99c6d5cc548a4be79a579760\": container with ID starting with a2b1027cc6cd3d31f9fba667369555c70f17053c99c6d5cc548a4be79a579760 not found: ID does not exist" Oct 30 12:46:10.568537 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.568506 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr"] Oct 30 12:46:10.572853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:10.572827 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ecb3aa/payload-processing-external-model-e2e-extmod-b-ecb3aa-5c45gc9mr"] Oct 30 12:46:11.032641 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:11.032606 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fa60d02-3a82-4f79-9382-bc3a84180c08" path="/var/lib/kubelet/pods/7fa60d02-3a82-4f79-9382-bc3a84180c08/volumes" Oct 30 12:46:11.186506 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:11.186470 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:12.476000 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:12.475969 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:12.645478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:12.645439 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:12.772134 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:12.772050 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:12.891741 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:12.891708 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:14.519006 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:14.518974 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:16.172401 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:16.172370 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:17.851419 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:17.851384 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:19.004537 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:19.004504 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:19.305380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:19.305307 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:19.498376 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:19.498339 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:19.706456 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:19.706422 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:19.853909 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:19.853875 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:20.017183 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:20.017109 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:20.800288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:20.800165 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:20.995154 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:20.988875 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:22.255300 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:22.255244 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:23.149168 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:23.149075 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:23.304541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:23.304503 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:23.518053 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:23.517969 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:23.722146 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:23.722118 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:23.839363 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:23.836819 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:24.259541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:24.259510 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:25.047050 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:25.047018 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:26.043588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:26.043549 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:26.410231 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:26.410197 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:26.904580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:26.904509 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:27.023900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:27.023840 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:27.657221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:27.657105 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:29.309824 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:29.309791 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:30.527972 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:30.527925 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:30.660759 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:30.660727 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:30.919784 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:30.919754 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7lb7d" Oct 30 12:46:31.044841 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:31.044809 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:31.284651 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:31.284566 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:32.217673 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.217638 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:32.348154 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.348121 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:32.394549 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.394520 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86"] Oct 30 12:46:32.395292 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395278 4413 cpu_manager.go:414] "containerMap: removing container" podUID="5ad807e4-0d66-4856-964a-5b7a4a3a80bb" containerName="payload-processing" Oct 30 12:46:32.395347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395294 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad807e4-0d66-4856-964a-5b7a4a3a80bb" containerName="payload-processing" Oct 30 12:46:32.395347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395308 4413 cpu_manager.go:414] "containerMap: removing container" podUID="e3b4a2cf-88e0-43af-a20d-f8d79f76368c" containerName="cleanup" Oct 30 12:46:32.395347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395313 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b4a2cf-88e0-43af-a20d-f8d79f76368c" containerName="cleanup" Oct 30 12:46:32.395347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395341 4413 cpu_manager.go:414] "containerMap: removing container" podUID="7d4b98ce-81d9-4aff-9c27-7c27119fc440" containerName="cleanup" Oct 30 12:46:32.395347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395346 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d4b98ce-81d9-4aff-9c27-7c27119fc440" containerName="cleanup" Oct 30 12:46:32.395512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395356 4413 cpu_manager.go:414] "containerMap: removing container" podUID="614cb5f4-97b0-46f2-8d23-c50b5b673eab" containerName="cleanup" Oct 30 12:46:32.395512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395362 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="614cb5f4-97b0-46f2-8d23-c50b5b673eab" containerName="cleanup" Oct 30 12:46:32.395512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395370 4413 cpu_manager.go:414] "containerMap: removing container" podUID="4cd33196-cd20-4832-9cb0-53063fb91c11" containerName="payload-processing" Oct 30 12:46:32.395512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395375 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd33196-cd20-4832-9cb0-53063fb91c11" containerName="payload-processing" Oct 30 12:46:32.395512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395387 4413 cpu_manager.go:414] "containerMap: removing container" podUID="9cb3d669-0630-425d-926f-fdfedac04d40" containerName="cleanup" Oct 30 12:46:32.395512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395391 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb3d669-0630-425d-926f-fdfedac04d40" containerName="cleanup" Oct 30 12:46:32.395512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395398 4413 cpu_manager.go:414] "containerMap: removing container" podUID="7fa60d02-3a82-4f79-9382-bc3a84180c08" containerName="payload-processing" Oct 30 12:46:32.395512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395404 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa60d02-3a82-4f79-9382-bc3a84180c08" containerName="payload-processing" Oct 30 12:46:32.395512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395514 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd33196-cd20-4832-9cb0-53063fb91c11" containerName="payload-processing" Oct 30 12:46:32.395788 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395523 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb3d669-0630-425d-926f-fdfedac04d40" containerName="cleanup" Oct 30 12:46:32.395788 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395530 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa60d02-3a82-4f79-9382-bc3a84180c08" containerName="payload-processing" Oct 30 12:46:32.395788 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395536 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="614cb5f4-97b0-46f2-8d23-c50b5b673eab" containerName="cleanup" Oct 30 12:46:32.395788 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395546 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b4a2cf-88e0-43af-a20d-f8d79f76368c" containerName="cleanup" Oct 30 12:46:32.395788 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.395554 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad807e4-0d66-4856-964a-5b7a4a3a80bb" containerName="payload-processing" Oct 30 12:46:32.400831 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.400815 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" Oct 30 12:46:32.417848 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.417824 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86"] Oct 30 12:46:32.425432 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.425409 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-fddaa3\"" Oct 30 12:46:32.425588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.425565 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-fddaa3-dockercfg-jw6gf\"" Oct 30 12:46:32.487059 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.486947 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2trv\" (UniqueName: \"kubernetes.io/projected/d3832da5-41b0-4993-b87d-971b41984eb3-kube-api-access-w2trv\") pod \"payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86\" (UID: \"d3832da5-41b0-4993-b87d-971b41984eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" Oct 30 12:46:32.487691 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.487665 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3832da5-41b0-4993-b87d-971b41984eb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86\" (UID: \"d3832da5-41b0-4993-b87d-971b41984eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" Oct 30 12:46:32.530134 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.530096 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj"] Oct 30 12:46:32.535134 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.535038 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" Oct 30 12:46:32.582929 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.582894 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj"] Oct 30 12:46:32.589063 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.589025 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2trv\" (UniqueName: \"kubernetes.io/projected/d3832da5-41b0-4993-b87d-971b41984eb3-kube-api-access-w2trv\") pod \"payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86\" (UID: \"d3832da5-41b0-4993-b87d-971b41984eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" Oct 30 12:46:32.589296 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.589276 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3832da5-41b0-4993-b87d-971b41984eb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86\" (UID: \"d3832da5-41b0-4993-b87d-971b41984eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" Oct 30 12:46:32.590103 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.590079 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3832da5-41b0-4993-b87d-971b41984eb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86\" (UID: \"d3832da5-41b0-4993-b87d-971b41984eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" Oct 30 12:46:32.620732 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.620697 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2trv\" (UniqueName: \"kubernetes.io/projected/d3832da5-41b0-4993-b87d-971b41984eb3-kube-api-access-w2trv\") pod \"payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86\" (UID: \"d3832da5-41b0-4993-b87d-971b41984eb3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" Oct 30 12:46:32.691021 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.690985 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf9sr\" (UniqueName: \"kubernetes.io/projected/99b68128-c265-4be3-a666-d3e73ad7fa39-kube-api-access-xf9sr\") pod \"payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj\" (UID: \"99b68128-c265-4be3-a666-d3e73ad7fa39\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" Oct 30 12:46:32.691216 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.691140 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99b68128-c265-4be3-a666-d3e73ad7fa39-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj\" (UID: \"99b68128-c265-4be3-a666-d3e73ad7fa39\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" Oct 30 12:46:32.715464 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.715438 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" Oct 30 12:46:32.792293 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.792173 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf9sr\" (UniqueName: \"kubernetes.io/projected/99b68128-c265-4be3-a666-d3e73ad7fa39-kube-api-access-xf9sr\") pod \"payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj\" (UID: \"99b68128-c265-4be3-a666-d3e73ad7fa39\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" Oct 30 12:46:32.792440 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.792297 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99b68128-c265-4be3-a666-d3e73ad7fa39-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj\" (UID: \"99b68128-c265-4be3-a666-d3e73ad7fa39\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" Oct 30 12:46:32.793232 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.792947 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99b68128-c265-4be3-a666-d3e73ad7fa39-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj\" (UID: \"99b68128-c265-4be3-a666-d3e73ad7fa39\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" Oct 30 12:46:32.816167 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.816025 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf9sr\" (UniqueName: \"kubernetes.io/projected/99b68128-c265-4be3-a666-d3e73ad7fa39-kube-api-access-xf9sr\") pod \"payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj\" (UID: \"99b68128-c265-4be3-a666-d3e73ad7fa39\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" Oct 30 12:46:32.857736 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:32.857698 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" Oct 30 12:46:33.025618 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:33.025552 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86"] Oct 30 12:46:33.025618 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:46:33.025567 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3832da5_41b0_4993_b87d_971b41984eb3.slice/crio-c16aa38714f14891e17688b3a564c2d703634b8944354f309854195a3662086e WatchSource:0}: Error finding container c16aa38714f14891e17688b3a564c2d703634b8944354f309854195a3662086e: Status 404 returned error can't find the container with id c16aa38714f14891e17688b3a564c2d703634b8944354f309854195a3662086e Oct 30 12:46:33.163020 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:33.162989 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj"] Oct 30 12:46:33.163365 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:46:33.163340 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b68128_c265_4be3_a666_d3e73ad7fa39.slice/crio-172a9de5ca88320a1d62cb94788aa836d8156931f180de374dcae0e89a7b9c49 WatchSource:0}: Error finding container 172a9de5ca88320a1d62cb94788aa836d8156931f180de374dcae0e89a7b9c49: Status 404 returned error can't find the container with id 172a9de5ca88320a1d62cb94788aa836d8156931f180de374dcae0e89a7b9c49 Oct 30 12:46:33.236769 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:33.236718 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:33.415302 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:33.415230 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:33.528769 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:33.528741 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:33.683305 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:33.683184 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" event={"ID":"99b68128-c265-4be3-a666-d3e73ad7fa39","Type":"ContainerStarted","Data":"b579fb1e4ec28095fa019f6c88664f5f513df6b4bfb53b5695247a1728b9bcf0"} Oct 30 12:46:33.683305 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:33.683266 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" event={"ID":"99b68128-c265-4be3-a666-d3e73ad7fa39","Type":"ContainerStarted","Data":"172a9de5ca88320a1d62cb94788aa836d8156931f180de374dcae0e89a7b9c49"} Oct 30 12:46:33.683533 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:33.683429 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" Oct 30 12:46:33.685135 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:33.685108 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" event={"ID":"d3832da5-41b0-4993-b87d-971b41984eb3","Type":"ContainerStarted","Data":"15d39e98ed09e8be209cf62513f65bb624b8cc4a7928ab679c478a004e9d1a7b"} Oct 30 12:46:33.685298 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:33.685149 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" event={"ID":"d3832da5-41b0-4993-b87d-971b41984eb3","Type":"ContainerStarted","Data":"c16aa38714f14891e17688b3a564c2d703634b8944354f309854195a3662086e"} Oct 30 12:46:33.685298 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:33.685238 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" Oct 30 12:46:33.724002 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:33.723951 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" podStartSLOduration=1.7239379910000001 podStartE2EDuration="1.723937991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:33.722472963 +0000 UTC m=+3146.832315873" watchObservedRunningTime="2026-10-30 12:46:33.723937991 +0000 UTC m=+3146.833780871" Oct 30 12:46:33.765310 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:33.765235 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" podStartSLOduration=1.765221554 podStartE2EDuration="1.765221554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:33.761671175 +0000 UTC m=+3146.871514056" watchObservedRunningTime="2026-10-30 12:46:33.765221554 +0000 UTC m=+3146.875064434" Oct 30 12:46:34.966311 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:34.966280 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:35.815467 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:46:35.815432 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:46:36.614444 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:36.614411 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:38.180207 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:38.180159 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:39.894332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:39.894300 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:41.456266 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:41.456170 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:43.083003 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:43.082885 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:44.695482 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:44.695455 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" Oct 30 12:46:44.695878 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:44.695524 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" Oct 30 12:46:44.770082 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:44.770042 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:46.370028 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:46.370000 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:48.096433 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:48.096398 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:49.863720 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:49.863683 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:51.538960 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:51.538929 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:53.163789 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:53.163748 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:54.913221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:54.913185 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:56.054186 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:56.054129 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:56.180326 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:56.180296 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:56.781218 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:56.781184 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:58.248057 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.248018 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:58.434092 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.434063 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6"] Oct 30 12:46:58.438852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.438836 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" Oct 30 12:46:58.457454 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.457424 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6"] Oct 30 12:46:58.464780 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.464744 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ad1da\"/\"agc-cred-21df89c992f6\"" Oct 30 12:46:58.464780 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.464760 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ad1da\"/\"payload-processing-external-model-e2e--da1b9d00-dockercfg-p948v\"" Oct 30 12:46:58.464989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.464790 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ad1da\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6ad1da\"" Oct 30 12:46:58.464989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.464812 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ad1da\"/\"openshift-service-ca.crt\"" Oct 30 12:46:58.465189 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.465156 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ad1da\"/\"routing-overlay\"" Oct 30 12:46:58.470286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.470242 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ad1da\"/\"kube-root-ca.crt\"" Oct 30 12:46:58.471107 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.471079 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:46:58.504916 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.504849 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfxcm\" (UniqueName: \"kubernetes.io/projected/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-kube-api-access-xfxcm\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6\" (UID: \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" Oct 30 12:46:58.505065 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.505046 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6\" (UID: \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" Oct 30 12:46:58.505111 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.505073 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6\" (UID: \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" Oct 30 12:46:58.505193 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.505182 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6\" (UID: \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" Oct 30 12:46:58.608934 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.608876 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfxcm\" (UniqueName: \"kubernetes.io/projected/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-kube-api-access-xfxcm\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6\" (UID: \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" Oct 30 12:46:58.610289 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.610262 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6\" (UID: \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" Oct 30 12:46:58.610491 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.610474 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6\" (UID: \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" Oct 30 12:46:58.610745 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.610729 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6\" (UID: \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" Oct 30 12:46:58.611268 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.611230 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6\" (UID: \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" Oct 30 12:46:58.612054 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.612029 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6\" (UID: \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" Oct 30 12:46:58.641301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.632798 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6\" (UID: \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" Oct 30 12:46:58.641301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.637360 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfxcm\" (UniqueName: \"kubernetes.io/projected/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-kube-api-access-xfxcm\") pod \"payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6\" (UID: \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\") " pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" Oct 30 12:46:58.754479 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:58.754438 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" Oct 30 12:46:59.066919 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:46:59.066893 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e8c4a3_4afb_4869_a24c_24e0dabf9a85.slice/crio-a883440a1a6fda1bbca06ac54f137f87dc2335f943802d5d9cdf4576f892269e WatchSource:0}: Error finding container a883440a1a6fda1bbca06ac54f137f87dc2335f943802d5d9cdf4576f892269e: Status 404 returned error can't find the container with id a883440a1a6fda1bbca06ac54f137f87dc2335f943802d5d9cdf4576f892269e Oct 30 12:46:59.067284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:59.067244 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6"] Oct 30 12:46:59.903619 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:59.903581 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" event={"ID":"61e8c4a3-4afb-4869-a24c-24e0dabf9a85","Type":"ContainerStarted","Data":"9fc9911d6b03aba8bd7ce2a339bd4c9181732fcba14c415ff3a60637de213273"} Oct 30 12:46:59.904066 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:59.903645 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" event={"ID":"61e8c4a3-4afb-4869-a24c-24e0dabf9a85","Type":"ContainerStarted","Data":"a883440a1a6fda1bbca06ac54f137f87dc2335f943802d5d9cdf4576f892269e"} Oct 30 12:46:59.904066 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:59.903674 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" Oct 30 12:46:59.944178 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:46:59.944135 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" podStartSLOduration=1.9441215 podStartE2EDuration="1.9441215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:59.943418608 +0000 UTC m=+3173.053261490" watchObservedRunningTime="2026-10-30 12:46:59.9441215 +0000 UTC m=+3173.053964380" Oct 30 12:47:00.149136 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:00.149102 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:00.282415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:00.282317 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:01.763661 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:01.763626 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:03.333529 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:03.333489 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:03.477526 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:03.477493 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:03.616017 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:03.615889 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:04.275157 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:04.275123 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6"] Oct 30 12:47:04.278816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:04.278795 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" Oct 30 12:47:04.352974 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:04.352922 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:05.240638 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:05.240604 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:05.815618 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:47:05.815580 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:47:06.872673 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:06.872635 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:08.478989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:08.478948 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:09.275563 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.275523 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" podUID="61e8c4a3-4afb-4869-a24c-24e0dabf9a85" containerName="payload-processing" containerID="cri-o://9fc9911d6b03aba8bd7ce2a339bd4c9181732fcba14c415ff3a60637de213273" gracePeriod=25 Oct 30 12:47:09.564122 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.564096 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" Oct 30 12:47:09.590275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.590227 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-plugins-config-volume\") pod \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\" (UID: \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\") " Oct 30 12:47:09.590421 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.590329 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-routing-overlay-volume\") pod \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\" (UID: \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\") " Oct 30 12:47:09.590421 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.590381 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-provider-credentials\" (UniqueName: \"kubernetes.io/projected/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-provider-credentials\") pod \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\" (UID: \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\") " Oct 30 12:47:09.590587 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.590568 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-kube-api-access-xfxcm\" (UniqueName: \"kubernetes.io/projected/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-kube-api-access-xfxcm\") pod \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\" (UID: \"61e8c4a3-4afb-4869-a24c-24e0dabf9a85\") " Oct 30 12:47:09.590656 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.590606 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-plugins-config-volume" pod "61e8c4a3-4afb-4869-a24c-24e0dabf9a85" (UID: "61e8c4a3-4afb-4869-a24c-24e0dabf9a85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:47:09.590748 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.590729 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-routing-overlay-volume" pod "61e8c4a3-4afb-4869-a24c-24e0dabf9a85" (UID: "61e8c4a3-4afb-4869-a24c-24e0dabf9a85"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:47:09.591175 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.591155 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:47:09.591175 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.591176 4413 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-routing-overlay-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:47:09.592850 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.592828 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-kube-api-access-xfxcm" pod "61e8c4a3-4afb-4869-a24c-24e0dabf9a85" (UID: "61e8c4a3-4afb-4869-a24c-24e0dabf9a85"). InnerVolumeSpecName "kube-api-access-xfxcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:47:09.592952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.592893 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-provider-credentials" pod "61e8c4a3-4afb-4869-a24c-24e0dabf9a85" (UID: "61e8c4a3-4afb-4869-a24c-24e0dabf9a85"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:47:09.692225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.692191 4413 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-provider-credentials\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:47:09.692225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.692219 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfxcm\" (UniqueName: \"kubernetes.io/projected/61e8c4a3-4afb-4869-a24c-24e0dabf9a85-kube-api-access-xfxcm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:47:09.976603 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.976566 4413 generic.go:356] "Generic (PLEG): container finished" podID="61e8c4a3-4afb-4869-a24c-24e0dabf9a85" containerID="9fc9911d6b03aba8bd7ce2a339bd4c9181732fcba14c415ff3a60637de213273" exitCode=0 Oct 30 12:47:09.976812 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.976619 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" Oct 30 12:47:09.976812 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.976652 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" event={"ID":"61e8c4a3-4afb-4869-a24c-24e0dabf9a85","Type":"ContainerDied","Data":"9fc9911d6b03aba8bd7ce2a339bd4c9181732fcba14c415ff3a60637de213273"} Oct 30 12:47:09.976812 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.976724 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6" event={"ID":"61e8c4a3-4afb-4869-a24c-24e0dabf9a85","Type":"ContainerDied","Data":"a883440a1a6fda1bbca06ac54f137f87dc2335f943802d5d9cdf4576f892269e"} Oct 30 12:47:09.976812 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.976755 4413 scope.go:122] "RemoveContainer" containerID="9fc9911d6b03aba8bd7ce2a339bd4c9181732fcba14c415ff3a60637de213273" Oct 30 12:47:09.989383 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.989364 4413 scope.go:122] "RemoveContainer" containerID="9fc9911d6b03aba8bd7ce2a339bd4c9181732fcba14c415ff3a60637de213273" Oct 30 12:47:09.989684 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:47:09.989653 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc9911d6b03aba8bd7ce2a339bd4c9181732fcba14c415ff3a60637de213273\": container with ID starting with 9fc9911d6b03aba8bd7ce2a339bd4c9181732fcba14c415ff3a60637de213273 not found: ID does not exist" containerID="9fc9911d6b03aba8bd7ce2a339bd4c9181732fcba14c415ff3a60637de213273" Oct 30 12:47:09.989752 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:09.989684 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc9911d6b03aba8bd7ce2a339bd4c9181732fcba14c415ff3a60637de213273"} err="failed to get container status \"9fc9911d6b03aba8bd7ce2a339bd4c9181732fcba14c415ff3a60637de213273\": rpc error: code = NotFound desc = could not find container \"9fc9911d6b03aba8bd7ce2a339bd4c9181732fcba14c415ff3a60637de213273\": container with ID starting with 9fc9911d6b03aba8bd7ce2a339bd4c9181732fcba14c415ff3a60637de213273 not found: ID does not exist" Oct 30 12:47:10.035313 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:10.035282 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6"] Oct 30 12:47:10.041530 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:10.041508 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ad1da/payload-processing-external-model-e2e-extmod-a-6ad1da-d874n9gz6"] Oct 30 12:47:10.128936 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:10.128903 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:11.022631 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:11.022602 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61e8c4a3-4afb-4869-a24c-24e0dabf9a85" path="/var/lib/kubelet/pods/61e8c4a3-4afb-4869-a24c-24e0dabf9a85/volumes" Oct 30 12:47:11.815259 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:11.815222 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:11.929223 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:11.929189 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:12.125135 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:12.125108 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:13.633422 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:13.633384 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:15.264152 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:15.264116 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:16.934671 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:16.934609 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:17.413229 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:17.413197 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:17.522542 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:17.522500 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:17.762456 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:17.762376 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68"] Oct 30 12:47:17.763179 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:17.763163 4413 cpu_manager.go:414] "containerMap: removing container" podUID="61e8c4a3-4afb-4869-a24c-24e0dabf9a85" containerName="payload-processing" Oct 30 12:47:17.763234 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:17.763180 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61e8c4a3-4afb-4869-a24c-24e0dabf9a85" containerName="payload-processing" Oct 30 12:47:17.763328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:17.763318 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="61e8c4a3-4afb-4869-a24c-24e0dabf9a85" containerName="payload-processing" Oct 30 12:47:17.768393 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:17.768374 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" Oct 30 12:47:17.786912 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:17.786881 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68"] Oct 30 12:47:17.792727 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:17.792705 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"agc-cred-d7c4ac25d5ef\"" Oct 30 12:47:17.792972 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:17.792952 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"payload-processing-external-model-e2e--d115ed06-dockercfg-4k7rq\"" Oct 30 12:47:17.793047 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:17.793000 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"openshift-service-ca.crt\"" Oct 30 12:47:17.793118 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:17.793086 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"routing-overlay\"" Oct 30 12:47:17.794578 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:17.794484 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"kube-root-ca.crt\"" Oct 30 12:47:17.794578 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:17.794553 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0de4b9\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0de4b9\"" Oct 30 12:47:17.932979 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:17.932936 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m25g8\" (UniqueName: \"kubernetes.io/projected/a5f02352-ca14-4f34-922f-cccbbfc011ea-kube-api-access-m25g8\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68\" (UID: \"a5f02352-ca14-4f34-922f-cccbbfc011ea\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" Oct 30 12:47:17.933164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:17.933021 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f02352-ca14-4f34-922f-cccbbfc011ea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68\" (UID: \"a5f02352-ca14-4f34-922f-cccbbfc011ea\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" Oct 30 12:47:17.933164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:17.933049 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f02352-ca14-4f34-922f-cccbbfc011ea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68\" (UID: \"a5f02352-ca14-4f34-922f-cccbbfc011ea\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" Oct 30 12:47:17.933240 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:17.933179 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f02352-ca14-4f34-922f-cccbbfc011ea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68\" (UID: \"a5f02352-ca14-4f34-922f-cccbbfc011ea\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" Oct 30 12:47:18.050332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:18.049274 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m25g8\" (UniqueName: \"kubernetes.io/projected/a5f02352-ca14-4f34-922f-cccbbfc011ea-kube-api-access-m25g8\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68\" (UID: \"a5f02352-ca14-4f34-922f-cccbbfc011ea\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" Oct 30 12:47:18.050332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:18.049338 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f02352-ca14-4f34-922f-cccbbfc011ea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68\" (UID: \"a5f02352-ca14-4f34-922f-cccbbfc011ea\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" Oct 30 12:47:18.050332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:18.049366 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f02352-ca14-4f34-922f-cccbbfc011ea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68\" (UID: \"a5f02352-ca14-4f34-922f-cccbbfc011ea\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" Oct 30 12:47:18.050332 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:18.049460 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f02352-ca14-4f34-922f-cccbbfc011ea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68\" (UID: \"a5f02352-ca14-4f34-922f-cccbbfc011ea\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" Oct 30 12:47:18.050995 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:18.050424 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f02352-ca14-4f34-922f-cccbbfc011ea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68\" (UID: \"a5f02352-ca14-4f34-922f-cccbbfc011ea\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" Oct 30 12:47:18.055713 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:18.051377 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f02352-ca14-4f34-922f-cccbbfc011ea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68\" (UID: \"a5f02352-ca14-4f34-922f-cccbbfc011ea\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" Oct 30 12:47:18.058168 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:18.058138 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f02352-ca14-4f34-922f-cccbbfc011ea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68\" (UID: \"a5f02352-ca14-4f34-922f-cccbbfc011ea\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" Oct 30 12:47:18.083156 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:18.083103 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m25g8\" (UniqueName: \"kubernetes.io/projected/a5f02352-ca14-4f34-922f-cccbbfc011ea-kube-api-access-m25g8\") pod \"payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68\" (UID: \"a5f02352-ca14-4f34-922f-cccbbfc011ea\") " pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" Oct 30 12:47:18.083567 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:18.083521 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" Oct 30 12:47:18.395953 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:47:18.395928 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f02352_ca14_4f34_922f_cccbbfc011ea.slice/crio-f89f7935d7fe37900ebf6cdcee3bf0190609bc5382767ab6b8d7b205219ac144 WatchSource:0}: Error finding container f89f7935d7fe37900ebf6cdcee3bf0190609bc5382767ab6b8d7b205219ac144: Status 404 returned error can't find the container with id f89f7935d7fe37900ebf6cdcee3bf0190609bc5382767ab6b8d7b205219ac144 Oct 30 12:47:18.396169 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:18.396148 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68"] Oct 30 12:47:18.671191 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:18.671118 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:19.060207 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:19.060100 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" event={"ID":"a5f02352-ca14-4f34-922f-cccbbfc011ea","Type":"ContainerStarted","Data":"c9117aa086ead8be81176ea39275eed7600576a2de89c2dfc9362c39fede63d3"} Oct 30 12:47:19.060207 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:19.060163 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" event={"ID":"a5f02352-ca14-4f34-922f-cccbbfc011ea","Type":"ContainerStarted","Data":"f89f7935d7fe37900ebf6cdcee3bf0190609bc5382767ab6b8d7b205219ac144"} Oct 30 12:47:19.060670 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:19.060210 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" Oct 30 12:47:19.100434 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:19.100386 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" podStartSLOduration=2.100373052 podStartE2EDuration="2.100373052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:19.10000715 +0000 UTC m=+3192.209850032" watchObservedRunningTime="2026-10-30 12:47:19.100373052 +0000 UTC m=+3192.210215932" Oct 30 12:47:21.124221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:21.124189 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:22.015120 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:22.015086 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:23.680726 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:23.680691 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:25.347351 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:25.347322 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:26.963192 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:26.963160 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:28.636706 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:28.636672 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:29.307420 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:29.307383 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68"] Oct 30 12:47:29.309986 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:29.309963 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" Oct 30 12:47:29.314708 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:29.314678 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:29.426778 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:29.426747 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:29.943279 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:29.943233 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:30.353666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:30.353569 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:30.472366 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:30.472153 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:30.640260 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:30.640225 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:31.012151 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:31.012042 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:31.163632 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:31.163600 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:31.366320 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:31.366224 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:31.550450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:31.550414 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:31.792664 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:31.792636 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:31.910521 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:31.910489 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:32.367480 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:32.367438 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:32.489227 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:32.489196 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:32.616453 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:32.616422 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:33.642759 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:33.642709 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:33.893355 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:33.893281 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c"] Oct 30 12:47:33.898517 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:33.898497 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" Oct 30 12:47:33.916642 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:33.916615 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c"] Oct 30 12:47:33.924719 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:33.924504 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2545c7\"" Oct 30 12:47:33.924719 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:33.924550 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"routing-overlay\"" Oct 30 12:47:33.924719 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:33.924582 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"openshift-service-ca.crt\"" Oct 30 12:47:33.924719 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:33.924683 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"kube-root-ca.crt\"" Oct 30 12:47:33.938667 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:33.938646 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"agc-cred-c4d4842908bc\"" Oct 30 12:47:33.938873 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:33.938651 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"payload-processing-external-model-e2e--03fcd75d-dockercfg-j74jr\"" Oct 30 12:47:34.016225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.015511 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc8d7\" (UniqueName: \"kubernetes.io/projected/652ba962-7a34-45c8-9736-292a2a3d3309-kube-api-access-tc8d7\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c\" (UID: \"652ba962-7a34-45c8-9736-292a2a3d3309\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" Oct 30 12:47:34.016225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.015614 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/652ba962-7a34-45c8-9736-292a2a3d3309-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c\" (UID: \"652ba962-7a34-45c8-9736-292a2a3d3309\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" Oct 30 12:47:34.016225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.015704 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/652ba962-7a34-45c8-9736-292a2a3d3309-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c\" (UID: \"652ba962-7a34-45c8-9736-292a2a3d3309\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" Oct 30 12:47:34.016225 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.015901 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/652ba962-7a34-45c8-9736-292a2a3d3309-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c\" (UID: \"652ba962-7a34-45c8-9736-292a2a3d3309\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" Oct 30 12:47:34.117646 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.117616 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/652ba962-7a34-45c8-9736-292a2a3d3309-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c\" (UID: \"652ba962-7a34-45c8-9736-292a2a3d3309\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" Oct 30 12:47:34.117853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.117672 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc8d7\" (UniqueName: \"kubernetes.io/projected/652ba962-7a34-45c8-9736-292a2a3d3309-kube-api-access-tc8d7\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c\" (UID: \"652ba962-7a34-45c8-9736-292a2a3d3309\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" Oct 30 12:47:34.117853 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.117749 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/652ba962-7a34-45c8-9736-292a2a3d3309-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c\" (UID: \"652ba962-7a34-45c8-9736-292a2a3d3309\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" Oct 30 12:47:34.117999 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.117852 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/652ba962-7a34-45c8-9736-292a2a3d3309-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c\" (UID: \"652ba962-7a34-45c8-9736-292a2a3d3309\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" Oct 30 12:47:34.118432 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.118399 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/652ba962-7a34-45c8-9736-292a2a3d3309-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c\" (UID: \"652ba962-7a34-45c8-9736-292a2a3d3309\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" Oct 30 12:47:34.118779 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.118752 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/652ba962-7a34-45c8-9736-292a2a3d3309-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c\" (UID: \"652ba962-7a34-45c8-9736-292a2a3d3309\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" Oct 30 12:47:34.120703 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.120673 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/652ba962-7a34-45c8-9736-292a2a3d3309-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c\" (UID: \"652ba962-7a34-45c8-9736-292a2a3d3309\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" Oct 30 12:47:34.140997 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.140973 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc8d7\" (UniqueName: \"kubernetes.io/projected/652ba962-7a34-45c8-9736-292a2a3d3309-kube-api-access-tc8d7\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c\" (UID: \"652ba962-7a34-45c8-9736-292a2a3d3309\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" Oct 30 12:47:34.161198 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.161144 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:34.214585 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.214554 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" Oct 30 12:47:34.308154 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.308110 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" podUID="a5f02352-ca14-4f34-922f-cccbbfc011ea" containerName="payload-processing" containerID="cri-o://c9117aa086ead8be81176ea39275eed7600576a2de89c2dfc9362c39fede63d3" gracePeriod=25 Oct 30 12:47:34.536529 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.536135 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c"] Oct 30 12:47:34.539073 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:47:34.539042 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652ba962_7a34_45c8_9736_292a2a3d3309.slice/crio-61634e49c18234adfd80e625652ced41346efb4cb5f5f225e4b8db9b040baa6b WatchSource:0}: Error finding container 61634e49c18234adfd80e625652ced41346efb4cb5f5f225e4b8db9b040baa6b: Status 404 returned error can't find the container with id 61634e49c18234adfd80e625652ced41346efb4cb5f5f225e4b8db9b040baa6b Oct 30 12:47:34.571477 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.571455 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" Oct 30 12:47:34.727521 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.727427 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f02352-ca14-4f34-922f-cccbbfc011ea-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f02352-ca14-4f34-922f-cccbbfc011ea-provider-credentials\") pod \"a5f02352-ca14-4f34-922f-cccbbfc011ea\" (UID: \"a5f02352-ca14-4f34-922f-cccbbfc011ea\") " Oct 30 12:47:34.727521 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.727498 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f02352-ca14-4f34-922f-cccbbfc011ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f02352-ca14-4f34-922f-cccbbfc011ea-plugins-config-volume\") pod \"a5f02352-ca14-4f34-922f-cccbbfc011ea\" (UID: \"a5f02352-ca14-4f34-922f-cccbbfc011ea\") " Oct 30 12:47:34.728006 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.727548 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f02352-ca14-4f34-922f-cccbbfc011ea-kube-api-access-m25g8\" (UniqueName: \"kubernetes.io/projected/a5f02352-ca14-4f34-922f-cccbbfc011ea-kube-api-access-m25g8\") pod \"a5f02352-ca14-4f34-922f-cccbbfc011ea\" (UID: \"a5f02352-ca14-4f34-922f-cccbbfc011ea\") " Oct 30 12:47:34.728006 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.727610 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f02352-ca14-4f34-922f-cccbbfc011ea-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f02352-ca14-4f34-922f-cccbbfc011ea-routing-overlay-volume\") pod \"a5f02352-ca14-4f34-922f-cccbbfc011ea\" (UID: \"a5f02352-ca14-4f34-922f-cccbbfc011ea\") " Oct 30 12:47:34.728006 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.727968 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f02352-ca14-4f34-922f-cccbbfc011ea-plugins-config-volume" pod "a5f02352-ca14-4f34-922f-cccbbfc011ea" (UID: "a5f02352-ca14-4f34-922f-cccbbfc011ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:47:34.728168 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.728025 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f02352-ca14-4f34-922f-cccbbfc011ea-routing-overlay-volume" pod "a5f02352-ca14-4f34-922f-cccbbfc011ea" (UID: "a5f02352-ca14-4f34-922f-cccbbfc011ea"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:47:34.728432 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.728382 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f02352-ca14-4f34-922f-cccbbfc011ea-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:47:34.728432 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.728400 4413 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f02352-ca14-4f34-922f-cccbbfc011ea-routing-overlay-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:47:34.729889 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.729863 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f02352-ca14-4f34-922f-cccbbfc011ea-kube-api-access-m25g8" pod "a5f02352-ca14-4f34-922f-cccbbfc011ea" (UID: "a5f02352-ca14-4f34-922f-cccbbfc011ea"). InnerVolumeSpecName "kube-api-access-m25g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:47:34.730019 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.729923 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f02352-ca14-4f34-922f-cccbbfc011ea-provider-credentials" pod "a5f02352-ca14-4f34-922f-cccbbfc011ea" (UID: "a5f02352-ca14-4f34-922f-cccbbfc011ea"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:47:34.830258 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.830208 4413 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f02352-ca14-4f34-922f-cccbbfc011ea-provider-credentials\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:47:34.830441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:34.830242 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m25g8\" (UniqueName: \"kubernetes.io/projected/a5f02352-ca14-4f34-922f-cccbbfc011ea-kube-api-access-m25g8\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:47:35.189049 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:35.189009 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" event={"ID":"652ba962-7a34-45c8-9736-292a2a3d3309","Type":"ContainerStarted","Data":"a5e8f12bf70dda2a77523ccbbf222381e95d14c85ce9736876e1080d5358b526"} Oct 30 12:47:35.189301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:35.189086 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" Oct 30 12:47:35.189301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:35.189109 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" event={"ID":"652ba962-7a34-45c8-9736-292a2a3d3309","Type":"ContainerStarted","Data":"61634e49c18234adfd80e625652ced41346efb4cb5f5f225e4b8db9b040baa6b"} Oct 30 12:47:35.191087 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:35.191053 4413 generic.go:356] "Generic (PLEG): container finished" podID="a5f02352-ca14-4f34-922f-cccbbfc011ea" containerID="c9117aa086ead8be81176ea39275eed7600576a2de89c2dfc9362c39fede63d3" exitCode=0 Oct 30 12:47:35.191214 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:35.191101 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" Oct 30 12:47:35.191214 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:35.191141 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" event={"ID":"a5f02352-ca14-4f34-922f-cccbbfc011ea","Type":"ContainerDied","Data":"c9117aa086ead8be81176ea39275eed7600576a2de89c2dfc9362c39fede63d3"} Oct 30 12:47:35.191214 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:35.191196 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68" event={"ID":"a5f02352-ca14-4f34-922f-cccbbfc011ea","Type":"ContainerDied","Data":"f89f7935d7fe37900ebf6cdcee3bf0190609bc5382767ab6b8d7b205219ac144"} Oct 30 12:47:35.191445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:35.191227 4413 scope.go:122] "RemoveContainer" containerID="c9117aa086ead8be81176ea39275eed7600576a2de89c2dfc9362c39fede63d3" Oct 30 12:47:35.204772 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:35.204752 4413 scope.go:122] "RemoveContainer" containerID="c9117aa086ead8be81176ea39275eed7600576a2de89c2dfc9362c39fede63d3" Oct 30 12:47:35.205064 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:47:35.205041 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9117aa086ead8be81176ea39275eed7600576a2de89c2dfc9362c39fede63d3\": container with ID starting with c9117aa086ead8be81176ea39275eed7600576a2de89c2dfc9362c39fede63d3 not found: ID does not exist" containerID="c9117aa086ead8be81176ea39275eed7600576a2de89c2dfc9362c39fede63d3" Oct 30 12:47:35.205155 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:35.205068 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9117aa086ead8be81176ea39275eed7600576a2de89c2dfc9362c39fede63d3"} err="failed to get container status \"c9117aa086ead8be81176ea39275eed7600576a2de89c2dfc9362c39fede63d3\": rpc error: code = NotFound desc = could not find container \"c9117aa086ead8be81176ea39275eed7600576a2de89c2dfc9362c39fede63d3\": container with ID starting with c9117aa086ead8be81176ea39275eed7600576a2de89c2dfc9362c39fede63d3 not found: ID does not exist" Oct 30 12:47:35.229869 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:35.229830 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" podStartSLOduration=2.229817612 podStartE2EDuration="2.229817612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:35.228340552 +0000 UTC m=+3208.338183433" watchObservedRunningTime="2026-10-30 12:47:35.229817612 +0000 UTC m=+3208.339660493" Oct 30 12:47:35.265982 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:35.265961 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68"] Oct 30 12:47:35.269315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:35.269295 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0de4b9/payload-processing-external-model-e2e-extmod-b-0de4b9-69b6kxn68"] Oct 30 12:47:35.814754 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:47:35.814718 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:47:35.833166 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:35.833139 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:36.448372 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:36.448334 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:36.560224 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:36.560197 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:36.673142 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:36.673109 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:37.022038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:37.022009 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f02352-ca14-4f34-922f-cccbbfc011ea" path="/var/lib/kubelet/pods/a5f02352-ca14-4f34-922f-cccbbfc011ea/volumes" Oct 30 12:47:37.474301 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:37.474269 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:39.278093 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:39.278062 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:40.958896 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:40.958854 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:41.374498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:41.374467 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:41.486630 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:41.486600 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:41.702433 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:41.702345 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:42.105219 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:42.105133 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:42.245551 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:42.245518 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:42.439136 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:42.439101 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:42.626775 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:42.626745 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:42.787663 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:42.787475 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:42.922666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:42.922633 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:43.496993 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:43.496965 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:43.611471 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:43.611441 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:44.435695 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:44.435659 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:44.569339 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:44.569279 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:44.720011 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:44.719899 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:44.863299 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:44.863268 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:45.002335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:45.002228 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:45.137202 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:45.137173 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:45.883775 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:45.883726 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:46.014006 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:46.013958 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:46.205528 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:46.205450 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" Oct 30 12:47:46.688895 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:46.688859 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:46.915400 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:46.915368 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:46.956150 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:46.956088 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2"] Oct 30 12:47:46.956846 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:46.956832 4413 cpu_manager.go:414] "containerMap: removing container" podUID="a5f02352-ca14-4f34-922f-cccbbfc011ea" containerName="payload-processing" Oct 30 12:47:46.956900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:46.956847 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f02352-ca14-4f34-922f-cccbbfc011ea" containerName="payload-processing" Oct 30 12:47:46.957006 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:46.956997 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f02352-ca14-4f34-922f-cccbbfc011ea" containerName="payload-processing" Oct 30 12:47:46.960791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:46.960767 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" Oct 30 12:47:46.975501 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:46.975476 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 12:47:46.975638 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:46.975550 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fedf2cef79c2\"" Oct 30 12:47:46.975712 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:46.975654 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 12:47:46.979398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:46.979377 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2"] Oct 30 12:47:46.987108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:46.987089 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 12:47:46.987458 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:46.987441 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-4ktzx\"" Oct 30 12:47:46.987565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:46.987494 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 12:47:47.019384 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:47.019355 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k4t5\" (UniqueName: \"kubernetes.io/projected/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-kube-api-access-5k4t5\") pod \"payload-processing-external-model-d5bdfd759-stjl2\" (UID: \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\") " pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" Oct 30 12:47:47.019540 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:47.019441 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-routing-overlay-volume\") pod \"payload-processing-external-model-d5bdfd759-stjl2\" (UID: \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\") " pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" Oct 30 12:47:47.019540 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:47.019518 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-plugins-config-volume\") pod \"payload-processing-external-model-d5bdfd759-stjl2\" (UID: \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\") " pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" Oct 30 12:47:47.019675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:47.019658 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-provider-credentials\") pod \"payload-processing-external-model-d5bdfd759-stjl2\" (UID: \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\") " pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" Oct 30 12:47:47.131272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:47.130415 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-provider-credentials\") pod \"payload-processing-external-model-d5bdfd759-stjl2\" (UID: \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\") " pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" Oct 30 12:47:47.131272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:47.130625 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k4t5\" (UniqueName: \"kubernetes.io/projected/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-kube-api-access-5k4t5\") pod \"payload-processing-external-model-d5bdfd759-stjl2\" (UID: \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\") " pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" Oct 30 12:47:47.131272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:47.130737 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-routing-overlay-volume\") pod \"payload-processing-external-model-d5bdfd759-stjl2\" (UID: \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\") " pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" Oct 30 12:47:47.131272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:47.130844 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-plugins-config-volume\") pod \"payload-processing-external-model-d5bdfd759-stjl2\" (UID: \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\") " pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" Oct 30 12:47:47.148827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:47.148797 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-provider-credentials\") pod \"payload-processing-external-model-d5bdfd759-stjl2\" (UID: \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\") " pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" Oct 30 12:47:47.150116 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:47.150090 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-routing-overlay-volume\") pod \"payload-processing-external-model-d5bdfd759-stjl2\" (UID: \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\") " pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" Oct 30 12:47:47.151369 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:47.151161 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-plugins-config-volume\") pod \"payload-processing-external-model-d5bdfd759-stjl2\" (UID: \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\") " pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" Oct 30 12:47:47.173062 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:47.173031 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k4t5\" (UniqueName: \"kubernetes.io/projected/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-kube-api-access-5k4t5\") pod \"payload-processing-external-model-d5bdfd759-stjl2\" (UID: \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\") " pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" Oct 30 12:47:47.275642 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:47.275558 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" Oct 30 12:47:47.587496 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:47.587463 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2"] Oct 30 12:47:47.600504 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:47:47.600473 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd3baac_0ccd_49a4_8ae5_cf0fa1385936.slice/crio-5c60807d6097336954600e406ca5de3ddc4405fa598017a4af2a7116ca7040d5 WatchSource:0}: Error finding container 5c60807d6097336954600e406ca5de3ddc4405fa598017a4af2a7116ca7040d5: Status 404 returned error can't find the container with id 5c60807d6097336954600e406ca5de3ddc4405fa598017a4af2a7116ca7040d5 Oct 30 12:47:48.314013 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:48.313980 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" event={"ID":"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936","Type":"ContainerStarted","Data":"5f6571fbf558db767074b2a03878e5659b280997ab59ffc6da23c0d6fb8005df"} Oct 30 12:47:48.314522 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:48.314027 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" event={"ID":"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936","Type":"ContainerStarted","Data":"5c60807d6097336954600e406ca5de3ddc4405fa598017a4af2a7116ca7040d5"} Oct 30 12:47:48.314522 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:48.314063 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" Oct 30 12:47:48.354906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:48.354861 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" podStartSLOduration=2.354847607 podStartE2EDuration="2.354847607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:48.353407769 +0000 UTC m=+3221.463250650" watchObservedRunningTime="2026-10-30 12:47:48.354847607 +0000 UTC m=+3221.464690529" Oct 30 12:47:48.545145 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:48.545114 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:50.339131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:50.339096 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:51.969884 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:51.969847 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:53.871144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:53.871100 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:55.634548 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:55.634505 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:56.150806 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:56.150761 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:56.269164 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:56.269133 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:57.274558 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:57.274527 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:57.387763 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:57.387723 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:57.501625 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:57.501590 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:47:59.325410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:59.325369 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" Oct 30 12:47:59.351982 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:47:59.351953 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:00.430244 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:00.430181 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:01.192233 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:01.192201 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:02.636472 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:02.636437 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:02.961535 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:02.961443 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:03.799782 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:03.799748 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:03.915377 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:03.915346 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:04.121555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:04.121458 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:04.390588 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:04.390547 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:04.512538 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:04.512487 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:04.644055 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:04.643959 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:04.781294 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:04.777417 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:05.152584 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:05.152545 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:05.275552 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:05.275517 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:05.814979 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:48:05.814948 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:48:06.364896 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:06.364852 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:08.010820 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:08.010782 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:09.636613 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:09.636581 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:11.362344 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:11.362309 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:12.165697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:12.165658 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:12.307146 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:12.307113 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:12.795614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:12.795584 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:13.214408 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:13.214375 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:14.762155 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:14.762119 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:16.370003 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:16.369966 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:17.929114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:17.929083 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:19.082466 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:19.082435 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:19.207962 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:19.207932 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:19.322842 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:19.322814 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:19.595048 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:19.595020 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:21.164748 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:21.164717 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:22.943681 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:22.943643 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:24.543297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:24.543234 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:24.690181 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:24.690151 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:24.808519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:24.808429 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:25.878762 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:25.878727 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:25.994097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:25.994057 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:26.203682 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:26.203607 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:26.885467 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:26.885435 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:27.007934 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:27.007906 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:28.397849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:28.397815 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:30.159487 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:30.159432 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:32.038629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:32.038599 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:34.044476 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:34.044440 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:35.760090 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:35.760055 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:35.824976 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:48:35.824940 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:48:37.402030 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:37.401993 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:38.981593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:38.981560 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:40.565748 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:40.565709 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:41.139963 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:41.139932 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2"] Oct 30 12:48:42.219512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:42.219478 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:43.879757 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:43.879726 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:45.527045 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:45.527018 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:46.140507 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.140473 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" podUID="3fd3baac-0ccd-49a4-8ae5-cf0fa1385936" containerName="payload-processing" containerID="cri-o://5f6571fbf558db767074b2a03878e5659b280997ab59ffc6da23c0d6fb8005df" gracePeriod=25 Oct 30 12:48:46.422734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.422709 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" Oct 30 12:48:46.532451 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.532420 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-kube-api-access-5k4t5\" (UniqueName: \"kubernetes.io/projected/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-kube-api-access-5k4t5\") pod \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\" (UID: \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\") " Oct 30 12:48:46.532894 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.532497 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-plugins-config-volume\") pod \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\" (UID: \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\") " Oct 30 12:48:46.532894 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.532533 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-routing-overlay-volume\") pod \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\" (UID: \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\") " Oct 30 12:48:46.532894 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.532719 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-provider-credentials\") pod \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\" (UID: \"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936\") " Oct 30 12:48:46.533089 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.532929 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-plugins-config-volume" pod "3fd3baac-0ccd-49a4-8ae5-cf0fa1385936" (UID: "3fd3baac-0ccd-49a4-8ae5-cf0fa1385936"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:48:46.533089 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.533004 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-routing-overlay-volume" pod "3fd3baac-0ccd-49a4-8ae5-cf0fa1385936" (UID: "3fd3baac-0ccd-49a4-8ae5-cf0fa1385936"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:48:46.533343 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.533323 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:48:46.533416 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.533347 4413 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-routing-overlay-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:48:46.535088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.535065 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-provider-credentials" pod "3fd3baac-0ccd-49a4-8ae5-cf0fa1385936" (UID: "3fd3baac-0ccd-49a4-8ae5-cf0fa1385936"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:48:46.535192 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.535176 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-kube-api-access-5k4t5" pod "3fd3baac-0ccd-49a4-8ae5-cf0fa1385936" (UID: "3fd3baac-0ccd-49a4-8ae5-cf0fa1385936"). InnerVolumeSpecName "kube-api-access-5k4t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:48:46.635127 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.635097 4413 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-provider-credentials\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:48:46.635127 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.635124 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k4t5\" (UniqueName: \"kubernetes.io/projected/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936-kube-api-access-5k4t5\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:48:46.819427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.819335 4413 generic.go:356] "Generic (PLEG): container finished" podID="3fd3baac-0ccd-49a4-8ae5-cf0fa1385936" containerID="5f6571fbf558db767074b2a03878e5659b280997ab59ffc6da23c0d6fb8005df" exitCode=0 Oct 30 12:48:46.819427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.819403 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" Oct 30 12:48:46.819677 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.819419 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" event={"ID":"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936","Type":"ContainerDied","Data":"5f6571fbf558db767074b2a03878e5659b280997ab59ffc6da23c0d6fb8005df"} Oct 30 12:48:46.819677 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.819530 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2" event={"ID":"3fd3baac-0ccd-49a4-8ae5-cf0fa1385936","Type":"ContainerDied","Data":"5c60807d6097336954600e406ca5de3ddc4405fa598017a4af2a7116ca7040d5"} Oct 30 12:48:46.819677 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.819563 4413 scope.go:122] "RemoveContainer" containerID="5f6571fbf558db767074b2a03878e5659b280997ab59ffc6da23c0d6fb8005df" Oct 30 12:48:46.830938 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.830921 4413 scope.go:122] "RemoveContainer" containerID="5f6571fbf558db767074b2a03878e5659b280997ab59ffc6da23c0d6fb8005df" Oct 30 12:48:46.831209 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:48:46.831191 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f6571fbf558db767074b2a03878e5659b280997ab59ffc6da23c0d6fb8005df\": container with ID starting with 5f6571fbf558db767074b2a03878e5659b280997ab59ffc6da23c0d6fb8005df not found: ID does not exist" containerID="5f6571fbf558db767074b2a03878e5659b280997ab59ffc6da23c0d6fb8005df" Oct 30 12:48:46.831284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.831217 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f6571fbf558db767074b2a03878e5659b280997ab59ffc6da23c0d6fb8005df"} err="failed to get container status \"5f6571fbf558db767074b2a03878e5659b280997ab59ffc6da23c0d6fb8005df\": rpc error: code = NotFound desc = could not find container \"5f6571fbf558db767074b2a03878e5659b280997ab59ffc6da23c0d6fb8005df\": container with ID starting with 5f6571fbf558db767074b2a03878e5659b280997ab59ffc6da23c0d6fb8005df not found: ID does not exist" Oct 30 12:48:46.872455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.872428 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2"] Oct 30 12:48:46.883551 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:46.883529 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5bdfd759-stjl2"] Oct 30 12:48:47.021662 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:47.021631 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fd3baac-0ccd-49a4-8ae5-cf0fa1385936" path="/var/lib/kubelet/pods/3fd3baac-0ccd-49a4-8ae5-cf0fa1385936/volumes" Oct 30 12:48:47.201088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:47.201055 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:48.811257 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:48.811224 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:50.535017 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:50.534988 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:52.259166 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:52.259126 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:54.310222 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:54.310188 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:55.880121 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:55.880089 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:57.562452 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:57.562421 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:48:59.194940 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:48:59.194904 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:00.544141 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:00.544103 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:00.778996 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:00.778966 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:01.785435 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:01.785287 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj"] Oct 30 12:49:01.787211 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:01.787184 4413 cpu_manager.go:414] "containerMap: removing container" podUID="3fd3baac-0ccd-49a4-8ae5-cf0fa1385936" containerName="payload-processing" Oct 30 12:49:01.787429 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:01.787414 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fd3baac-0ccd-49a4-8ae5-cf0fa1385936" containerName="payload-processing" Oct 30 12:49:01.787924 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:01.787902 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fd3baac-0ccd-49a4-8ae5-cf0fa1385936" containerName="payload-processing" Oct 30 12:49:01.799323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:01.798092 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:01.821910 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:01.819504 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj"] Oct 30 12:49:01.947266 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:01.947211 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:01.947472 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:01.947322 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe31c94d-c481-44ee-8183-296ce6090146-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:01.947547 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:01.947491 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:01.947547 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:01.947527 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rxww\" (UniqueName: \"kubernetes.io/projected/fe31c94d-c481-44ee-8183-296ce6090146-kube-api-access-2rxww\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:01.947652 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:01.947627 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe31c94d-c481-44ee-8183-296ce6090146-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:01.947711 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:01.947655 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:01.947711 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:01.947677 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe31c94d-c481-44ee-8183-296ce6090146-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:01.947816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:01.947769 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:01.947946 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:01.947931 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.049768 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.049675 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.049768 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.049711 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rxww\" (UniqueName: \"kubernetes.io/projected/fe31c94d-c481-44ee-8183-296ce6090146-kube-api-access-2rxww\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.049768 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.049749 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe31c94d-c481-44ee-8183-296ce6090146-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.049768 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.049767 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.050145 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.049789 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe31c94d-c481-44ee-8183-296ce6090146-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.050145 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.049830 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.050145 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.050020 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.050145 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.050116 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.050437 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.050153 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe31c94d-c481-44ee-8183-296ce6090146-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.050437 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.050199 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.050437 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.050230 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.050615 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.050491 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.050615 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.050528 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.050735 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.050704 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe31c94d-c481-44ee-8183-296ce6090146-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.052654 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.052619 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.052796 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.052775 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe31c94d-c481-44ee-8183-296ce6090146-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.073088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.073056 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe31c94d-c481-44ee-8183-296ce6090146-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.073232 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.073197 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rxww\" (UniqueName: \"kubernetes.io/projected/fe31c94d-c481-44ee-8183-296ce6090146-kube-api-access-2rxww\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-9ngvj\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.127586 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.127555 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:02.454729 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.454697 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj"] Oct 30 12:49:02.455231 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:49:02.455205 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe31c94d_c481_44ee_8183_296ce6090146.slice/crio-793ac5df9b9dc5b97ccc351528593dbef5f793cd5151bd2a9a7656bb46ef248b WatchSource:0}: Error finding container 793ac5df9b9dc5b97ccc351528593dbef5f793cd5151bd2a9a7656bb46ef248b: Status 404 returned error can't find the container with id 793ac5df9b9dc5b97ccc351528593dbef5f793cd5151bd2a9a7656bb46ef248b Oct 30 12:49:02.458240 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.458199 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:49:02.458377 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.458358 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:49:02.458424 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.458402 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:49:02.542453 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.542423 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:02.953961 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.953923 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" event={"ID":"fe31c94d-c481-44ee-8183-296ce6090146","Type":"ContainerStarted","Data":"5f6adfb5c49ebbafce9010d30fcd3f8c1b4beb05e90c3c69d7fec78eb3e06744"} Oct 30 12:49:02.954520 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:02.953983 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" event={"ID":"fe31c94d-c481-44ee-8183-296ce6090146","Type":"ContainerStarted","Data":"793ac5df9b9dc5b97ccc351528593dbef5f793cd5151bd2a9a7656bb46ef248b"} Oct 30 12:49:03.003318 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:03.003236 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" podStartSLOduration=2.00321523 podStartE2EDuration="2.00321523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:03.000480195 +0000 UTC m=+3296.110323079" watchObservedRunningTime="2026-10-30 12:49:03.00321523 +0000 UTC m=+3296.113058112" Oct 30 12:49:03.128634 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:03.128604 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:03.134273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:03.133376 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 12:49:03.134273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:03.133449 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" podUID="fe31c94d-c481-44ee-8183-296ce6090146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 12:49:04.127962 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:04.127926 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 12:49:04.128468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:04.128000 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" podUID="fe31c94d-c481-44ee-8183-296ce6090146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 12:49:04.422899 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:04.422790 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:05.128551 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:05.128514 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 12:49:05.128969 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:05.128577 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" podUID="fe31c94d-c481-44ee-8183-296ce6090146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 12:49:05.845586 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:49:05.845538 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:49:06.129557 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:06.129520 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 12:49:06.130163 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:06.130130 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" podUID="fe31c94d-c481-44ee-8183-296ce6090146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 12:49:06.388187 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:06.388100 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:07.128414 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:07.128375 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 12:49:07.128615 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:07.128436 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" podUID="fe31c94d-c481-44ee-8183-296ce6090146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 12:49:08.127906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:08.127869 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 12:49:08.128347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:08.127934 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" podUID="fe31c94d-c481-44ee-8183-296ce6090146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 12:49:08.230166 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:08.230132 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:09.128218 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:09.128180 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 12:49:09.128737 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:09.128230 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" podUID="fe31c94d-c481-44ee-8183-296ce6090146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 12:49:09.837015 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:09.836980 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:10.128833 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:10.128799 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 30 12:49:10.129323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:10.129287 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" podUID="fe31c94d-c481-44ee-8183-296ce6090146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 30 12:49:11.582002 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:11.581967 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:12.168427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:12.168383 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:49:12.168623 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:12.168452 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" podUID="fe31c94d-c481-44ee-8183-296ce6090146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:49:13.210432 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:13.210395 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:49:13.210954 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:13.210450 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" podUID="fe31c94d-c481-44ee-8183-296ce6090146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:49:13.248268 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:13.248221 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:13.248797 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:13.248775 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:13.254579 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:13.254552 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 12:49:13.262593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:13.262566 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:14.858604 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:14.858557 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:16.547174 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:16.547139 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:18.259999 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:18.259967 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:18.766578 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:18.766528 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:18.776114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:18.776089 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c"] Oct 30 12:49:18.900575 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:18.900544 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:19.367151 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:19.367118 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:20.047431 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:20.047402 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:21.679162 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:21.679133 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:23.350590 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:23.350508 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:23.776969 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:23.776926 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" podUID="652ba962-7a34-45c8-9736-292a2a3d3309" containerName="payload-processing" containerID="cri-o://a5e8f12bf70dda2a77523ccbbf222381e95d14c85ce9736876e1080d5358b526" gracePeriod=25 Oct 30 12:49:24.070553 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.070529 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" Oct 30 12:49:24.140195 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.140162 4413 generic.go:356] "Generic (PLEG): container finished" podID="652ba962-7a34-45c8-9736-292a2a3d3309" containerID="a5e8f12bf70dda2a77523ccbbf222381e95d14c85ce9736876e1080d5358b526" exitCode=0 Oct 30 12:49:24.140414 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.140227 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" Oct 30 12:49:24.140414 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.140406 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" event={"ID":"652ba962-7a34-45c8-9736-292a2a3d3309","Type":"ContainerDied","Data":"a5e8f12bf70dda2a77523ccbbf222381e95d14c85ce9736876e1080d5358b526"} Oct 30 12:49:24.140548 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.140457 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c" event={"ID":"652ba962-7a34-45c8-9736-292a2a3d3309","Type":"ContainerDied","Data":"61634e49c18234adfd80e625652ced41346efb4cb5f5f225e4b8db9b040baa6b"} Oct 30 12:49:24.140548 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.140489 4413 scope.go:122] "RemoveContainer" containerID="a5e8f12bf70dda2a77523ccbbf222381e95d14c85ce9736876e1080d5358b526" Oct 30 12:49:24.158011 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.157990 4413 scope.go:122] "RemoveContainer" containerID="a5e8f12bf70dda2a77523ccbbf222381e95d14c85ce9736876e1080d5358b526" Oct 30 12:49:24.158343 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:49:24.158320 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e8f12bf70dda2a77523ccbbf222381e95d14c85ce9736876e1080d5358b526\": container with ID starting with a5e8f12bf70dda2a77523ccbbf222381e95d14c85ce9736876e1080d5358b526 not found: ID does not exist" containerID="a5e8f12bf70dda2a77523ccbbf222381e95d14c85ce9736876e1080d5358b526" Oct 30 12:49:24.158445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.158357 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e8f12bf70dda2a77523ccbbf222381e95d14c85ce9736876e1080d5358b526"} err="failed to get container status \"a5e8f12bf70dda2a77523ccbbf222381e95d14c85ce9736876e1080d5358b526\": rpc error: code = NotFound desc = could not find container \"a5e8f12bf70dda2a77523ccbbf222381e95d14c85ce9736876e1080d5358b526\": container with ID starting with a5e8f12bf70dda2a77523ccbbf222381e95d14c85ce9736876e1080d5358b526 not found: ID does not exist" Oct 30 12:49:24.210017 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.209993 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652ba962-7a34-45c8-9736-292a2a3d3309-kube-api-access-tc8d7\" (UniqueName: \"kubernetes.io/projected/652ba962-7a34-45c8-9736-292a2a3d3309-kube-api-access-tc8d7\") pod \"652ba962-7a34-45c8-9736-292a2a3d3309\" (UID: \"652ba962-7a34-45c8-9736-292a2a3d3309\") " Oct 30 12:49:24.210170 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.210120 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/652ba962-7a34-45c8-9736-292a2a3d3309-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/652ba962-7a34-45c8-9736-292a2a3d3309-plugins-config-volume\") pod \"652ba962-7a34-45c8-9736-292a2a3d3309\" (UID: \"652ba962-7a34-45c8-9736-292a2a3d3309\") " Oct 30 12:49:24.210170 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.210159 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652ba962-7a34-45c8-9736-292a2a3d3309-provider-credentials\" (UniqueName: \"kubernetes.io/projected/652ba962-7a34-45c8-9736-292a2a3d3309-provider-credentials\") pod \"652ba962-7a34-45c8-9736-292a2a3d3309\" (UID: \"652ba962-7a34-45c8-9736-292a2a3d3309\") " Oct 30 12:49:24.210338 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.210190 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/652ba962-7a34-45c8-9736-292a2a3d3309-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/652ba962-7a34-45c8-9736-292a2a3d3309-routing-overlay-volume\") pod \"652ba962-7a34-45c8-9736-292a2a3d3309\" (UID: \"652ba962-7a34-45c8-9736-292a2a3d3309\") " Oct 30 12:49:24.210627 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.210602 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652ba962-7a34-45c8-9736-292a2a3d3309-routing-overlay-volume" pod "652ba962-7a34-45c8-9736-292a2a3d3309" (UID: "652ba962-7a34-45c8-9736-292a2a3d3309"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:49:24.210712 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.210607 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652ba962-7a34-45c8-9736-292a2a3d3309-plugins-config-volume" pod "652ba962-7a34-45c8-9736-292a2a3d3309" (UID: "652ba962-7a34-45c8-9736-292a2a3d3309"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:49:24.211335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.211304 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/652ba962-7a34-45c8-9736-292a2a3d3309-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:49:24.211454 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.211340 4413 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/652ba962-7a34-45c8-9736-292a2a3d3309-routing-overlay-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:49:24.212649 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.212630 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652ba962-7a34-45c8-9736-292a2a3d3309-kube-api-access-tc8d7" pod "652ba962-7a34-45c8-9736-292a2a3d3309" (UID: "652ba962-7a34-45c8-9736-292a2a3d3309"). InnerVolumeSpecName "kube-api-access-tc8d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:49:24.212729 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.212703 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652ba962-7a34-45c8-9736-292a2a3d3309-provider-credentials" pod "652ba962-7a34-45c8-9736-292a2a3d3309" (UID: "652ba962-7a34-45c8-9736-292a2a3d3309"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:49:24.313285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.313192 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc8d7\" (UniqueName: \"kubernetes.io/projected/652ba962-7a34-45c8-9736-292a2a3d3309-kube-api-access-tc8d7\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:49:24.313285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.313221 4413 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/652ba962-7a34-45c8-9736-292a2a3d3309-provider-credentials\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:49:24.486963 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.486924 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c"] Oct 30 12:49:24.498098 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:24.498073 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-76bclgj5c"] Oct 30 12:49:25.021509 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:25.021468 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="652ba962-7a34-45c8-9736-292a2a3d3309" path="/var/lib/kubelet/pods/652ba962-7a34-45c8-9736-292a2a3d3309/volumes" Oct 30 12:49:25.053666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:25.053633 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:25.619523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:25.619492 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:25.741628 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:25.741589 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:25.858971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:25.858908 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:26.745565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:26.745525 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:28.445654 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:28.445621 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:30.045367 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:30.045337 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:31.753030 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:31.752990 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:31.914792 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:31.914754 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz"] Oct 30 12:49:31.915863 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:31.915843 4413 cpu_manager.go:414] "containerMap: removing container" podUID="652ba962-7a34-45c8-9736-292a2a3d3309" containerName="payload-processing" Oct 30 12:49:31.915863 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:31.915864 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652ba962-7a34-45c8-9736-292a2a3d3309" containerName="payload-processing" Oct 30 12:49:31.916070 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:31.916055 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="652ba962-7a34-45c8-9736-292a2a3d3309" containerName="payload-processing" Oct 30 12:49:31.923007 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:31.922982 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:31.943911 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:31.943865 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz"] Oct 30 12:49:32.007055 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.006973 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8fc0118-77f2-457b-8a58-82882ee852d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.007055 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.007020 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.007275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.007110 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.007275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.007215 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.007359 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.007289 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g9fn\" (UniqueName: \"kubernetes.io/projected/b8fc0118-77f2-457b-8a58-82882ee852d0-kube-api-access-8g9fn\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.007416 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.007356 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.007514 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.007486 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.007590 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.007555 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.007709 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.007692 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.109296 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.109270 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.109467 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.109323 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.109467 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.109395 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.109554 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.109498 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8fc0118-77f2-457b-8a58-82882ee852d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.109554 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.109521 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.109862 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.109571 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.109862 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.109632 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.109862 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.109657 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g9fn\" (UniqueName: \"kubernetes.io/projected/b8fc0118-77f2-457b-8a58-82882ee852d0-kube-api-access-8g9fn\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.109862 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.109720 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.110288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.110238 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.110381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.110317 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.110575 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.110551 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.110677 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.110599 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.110677 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.110642 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8fc0118-77f2-457b-8a58-82882ee852d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.112048 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.112016 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.112163 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.112142 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.134950 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.134927 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.137874 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.137847 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g9fn\" (UniqueName: \"kubernetes.io/projected/b8fc0118-77f2-457b-8a58-82882ee852d0-kube-api-access-8g9fn\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-bc9xz\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.252170 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.252139 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:32.576156 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:49:32.576124 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8fc0118_77f2_457b_8a58_82882ee852d0.slice/crio-d86b1c4c4f0db48133955136431b0829fe8ed7e9197828edc83aaba0a112d716 WatchSource:0}: Error finding container d86b1c4c4f0db48133955136431b0829fe8ed7e9197828edc83aaba0a112d716: Status 404 returned error can't find the container with id d86b1c4c4f0db48133955136431b0829fe8ed7e9197828edc83aaba0a112d716 Oct 30 12:49:32.576414 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.576394 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz"] Oct 30 12:49:32.579010 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.578975 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:49:32.579137 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.579054 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:49:32.579137 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.579101 4413 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 12:49:32.591592 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.591567 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:32.716924 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:32.716886 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:33.210465 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:33.210434 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" event={"ID":"b8fc0118-77f2-457b-8a58-82882ee852d0","Type":"ContainerStarted","Data":"4a17b7d141409fc63ceb2e44294c9e5ceb466859a1bc9875c364c08d466ec74f"} Oct 30 12:49:33.210942 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:33.210481 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" event={"ID":"b8fc0118-77f2-457b-8a58-82882ee852d0","Type":"ContainerStarted","Data":"d86b1c4c4f0db48133955136431b0829fe8ed7e9197828edc83aaba0a112d716"} Oct 30 12:49:33.252477 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:33.252454 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:33.254070 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:33.254043 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 12:49:33.254199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:33.254086 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" podUID="b8fc0118-77f2-457b-8a58-82882ee852d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 12:49:33.285743 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:33.285697 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" podStartSLOduration=2.285685342 podStartE2EDuration="2.285685342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:33.272160032 +0000 UTC m=+3326.382002913" watchObservedRunningTime="2026-10-30 12:49:33.285685342 +0000 UTC m=+3326.395528222" Oct 30 12:49:33.443216 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:33.443183 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:33.562826 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:33.562737 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:33.710276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:33.708177 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:33.843478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:33.838749 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:34.253468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:34.253432 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 12:49:34.253908 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:34.253490 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" podUID="b8fc0118-77f2-457b-8a58-82882ee852d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 12:49:35.030988 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.030948 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm"] Oct 30 12:49:35.035852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.035821 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" Oct 30 12:49:35.049299 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.049263 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm"] Oct 30 12:49:35.053696 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.053668 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3088acb2-d284-4267-af59-7bbbef0bb012-plugins-config-volume\") pod \"payload-processing-external-model-64b8599cdb-bxjhm\" (UID: \"3088acb2-d284-4267-af59-7bbbef0bb012\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" Oct 30 12:49:35.053828 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.053747 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3088acb2-d284-4267-af59-7bbbef0bb012-provider-credentials\") pod \"payload-processing-external-model-64b8599cdb-bxjhm\" (UID: \"3088acb2-d284-4267-af59-7bbbef0bb012\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" Oct 30 12:49:35.053897 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.053831 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ffkl\" (UniqueName: \"kubernetes.io/projected/3088acb2-d284-4267-af59-7bbbef0bb012-kube-api-access-6ffkl\") pod \"payload-processing-external-model-64b8599cdb-bxjhm\" (UID: \"3088acb2-d284-4267-af59-7bbbef0bb012\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" Oct 30 12:49:35.053980 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.053961 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3088acb2-d284-4267-af59-7bbbef0bb012-routing-overlay-volume\") pod \"payload-processing-external-model-64b8599cdb-bxjhm\" (UID: \"3088acb2-d284-4267-af59-7bbbef0bb012\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" Oct 30 12:49:35.059560 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.059528 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 12:49:35.064163 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.064139 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-4ktzx\"" Oct 30 12:49:35.064322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.064306 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-643ff6d3028e\"" Oct 30 12:49:35.064410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.064373 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4ff758d7dea0\"" Oct 30 12:49:35.064515 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.064499 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fedf2cef79c2\"" Oct 30 12:49:35.064617 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.064602 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 12:49:35.064859 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.064839 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 12:49:35.064954 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.064935 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 12:49:35.155650 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.155616 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ffkl\" (UniqueName: \"kubernetes.io/projected/3088acb2-d284-4267-af59-7bbbef0bb012-kube-api-access-6ffkl\") pod \"payload-processing-external-model-64b8599cdb-bxjhm\" (UID: \"3088acb2-d284-4267-af59-7bbbef0bb012\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" Oct 30 12:49:35.155857 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.155744 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3088acb2-d284-4267-af59-7bbbef0bb012-routing-overlay-volume\") pod \"payload-processing-external-model-64b8599cdb-bxjhm\" (UID: \"3088acb2-d284-4267-af59-7bbbef0bb012\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" Oct 30 12:49:35.155961 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.155941 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3088acb2-d284-4267-af59-7bbbef0bb012-plugins-config-volume\") pod \"payload-processing-external-model-64b8599cdb-bxjhm\" (UID: \"3088acb2-d284-4267-af59-7bbbef0bb012\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" Oct 30 12:49:35.156030 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.155988 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3088acb2-d284-4267-af59-7bbbef0bb012-provider-credentials\") pod \"payload-processing-external-model-64b8599cdb-bxjhm\" (UID: \"3088acb2-d284-4267-af59-7bbbef0bb012\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" Oct 30 12:49:35.156524 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.156498 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3088acb2-d284-4267-af59-7bbbef0bb012-routing-overlay-volume\") pod \"payload-processing-external-model-64b8599cdb-bxjhm\" (UID: \"3088acb2-d284-4267-af59-7bbbef0bb012\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" Oct 30 12:49:35.156670 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.156620 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3088acb2-d284-4267-af59-7bbbef0bb012-plugins-config-volume\") pod \"payload-processing-external-model-64b8599cdb-bxjhm\" (UID: \"3088acb2-d284-4267-af59-7bbbef0bb012\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" Oct 30 12:49:35.158928 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.158906 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3088acb2-d284-4267-af59-7bbbef0bb012-provider-credentials\") pod \"payload-processing-external-model-64b8599cdb-bxjhm\" (UID: \"3088acb2-d284-4267-af59-7bbbef0bb012\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" Oct 30 12:49:35.178392 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.178373 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ffkl\" (UniqueName: \"kubernetes.io/projected/3088acb2-d284-4267-af59-7bbbef0bb012-kube-api-access-6ffkl\") pod \"payload-processing-external-model-64b8599cdb-bxjhm\" (UID: \"3088acb2-d284-4267-af59-7bbbef0bb012\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" Oct 30 12:49:35.253432 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.253398 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 12:49:35.253562 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.253456 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" podUID="b8fc0118-77f2-457b-8a58-82882ee852d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 12:49:35.365490 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.365414 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" Oct 30 12:49:35.721490 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.721457 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm"] Oct 30 12:49:35.721704 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:49:35.721564 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3088acb2_d284_4267_af59_7bbbef0bb012.slice/crio-618f7e091e5ec9131cefce637b385e9366efb9491ec27c44aa4a8113d009d10a WatchSource:0}: Error finding container 618f7e091e5ec9131cefce637b385e9366efb9491ec27c44aa4a8113d009d10a: Status 404 returned error can't find the container with id 618f7e091e5ec9131cefce637b385e9366efb9491ec27c44aa4a8113d009d10a Oct 30 12:49:35.748781 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:35.748752 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:35.861230 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:49:35.861194 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:49:36.240513 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:36.240473 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" event={"ID":"3088acb2-d284-4267-af59-7bbbef0bb012","Type":"ContainerStarted","Data":"f6dee50be2e21e4944b02b5406be1a97d61bf5614627a9802b528552a01a2c74"} Oct 30 12:49:36.240699 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:36.240530 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" event={"ID":"3088acb2-d284-4267-af59-7bbbef0bb012","Type":"ContainerStarted","Data":"618f7e091e5ec9131cefce637b385e9366efb9491ec27c44aa4a8113d009d10a"} Oct 30 12:49:36.240699 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:36.240567 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" Oct 30 12:49:36.252970 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:36.252940 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 12:49:36.253095 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:36.252987 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" podUID="b8fc0118-77f2-457b-8a58-82882ee852d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 12:49:36.281991 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:36.281929 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" podStartSLOduration=2.281918464 podStartE2EDuration="2.281918464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:36.280469591 +0000 UTC m=+3329.390312473" watchObservedRunningTime="2026-10-30 12:49:36.281918464 +0000 UTC m=+3329.391761385" Oct 30 12:49:37.252793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:37.252761 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 12:49:37.252984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:37.252811 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" podUID="b8fc0118-77f2-457b-8a58-82882ee852d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 12:49:37.481734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:37.481703 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:38.252673 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:38.252633 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 12:49:38.252892 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:38.252706 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" podUID="b8fc0118-77f2-457b-8a58-82882ee852d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 12:49:39.253111 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:39.253068 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 12:49:39.253603 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:39.253160 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" podUID="b8fc0118-77f2-457b-8a58-82882ee852d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 12:49:39.296108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:39.296070 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:39.769578 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:39.769545 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:39.885552 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:39.885514 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:40.108136 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:40.108058 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm"] Oct 30 12:49:40.111097 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:40.111075 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" Oct 30 12:49:40.252733 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:40.252701 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 12:49:40.252907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:40.252750 4413 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" podUID="b8fc0118-77f2-457b-8a58-82882ee852d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 12:49:40.948885 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:40.948851 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:41.086156 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:41.086122 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:41.292371 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:41.292270 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:41.793621 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:41.793579 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:41.794130 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:41.794106 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:41.795074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:41.795053 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 12:49:41.879213 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:41.879183 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls"] Oct 30 12:49:41.884033 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:41.884014 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" Oct 30 12:49:41.927724 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:41.927688 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls"] Oct 30 12:49:42.000903 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:42.000867 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e812649f-0314-4845-bc6c-8d7d4b225286-provider-credentials\") pod \"payload-processing-external-model-64b8599cdb-z7hls\" (UID: \"e812649f-0314-4845-bc6c-8d7d4b225286\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" Oct 30 12:49:42.001402 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:42.001108 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e812649f-0314-4845-bc6c-8d7d4b225286-plugins-config-volume\") pod \"payload-processing-external-model-64b8599cdb-z7hls\" (UID: \"e812649f-0314-4845-bc6c-8d7d4b225286\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" Oct 30 12:49:42.001402 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:42.001317 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e812649f-0314-4845-bc6c-8d7d4b225286-routing-overlay-volume\") pod \"payload-processing-external-model-64b8599cdb-z7hls\" (UID: \"e812649f-0314-4845-bc6c-8d7d4b225286\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" Oct 30 12:49:42.001524 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:42.001434 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7l9t\" (UniqueName: \"kubernetes.io/projected/e812649f-0314-4845-bc6c-8d7d4b225286-kube-api-access-t7l9t\") pod \"payload-processing-external-model-64b8599cdb-z7hls\" (UID: \"e812649f-0314-4845-bc6c-8d7d4b225286\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" Oct 30 12:49:42.103169 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:42.103096 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e812649f-0314-4845-bc6c-8d7d4b225286-provider-credentials\") pod \"payload-processing-external-model-64b8599cdb-z7hls\" (UID: \"e812649f-0314-4845-bc6c-8d7d4b225286\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" Oct 30 12:49:42.103369 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:42.103221 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e812649f-0314-4845-bc6c-8d7d4b225286-plugins-config-volume\") pod \"payload-processing-external-model-64b8599cdb-z7hls\" (UID: \"e812649f-0314-4845-bc6c-8d7d4b225286\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" Oct 30 12:49:42.103369 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:42.103325 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e812649f-0314-4845-bc6c-8d7d4b225286-routing-overlay-volume\") pod \"payload-processing-external-model-64b8599cdb-z7hls\" (UID: \"e812649f-0314-4845-bc6c-8d7d4b225286\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" Oct 30 12:49:42.103500 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:42.103371 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7l9t\" (UniqueName: \"kubernetes.io/projected/e812649f-0314-4845-bc6c-8d7d4b225286-kube-api-access-t7l9t\") pod \"payload-processing-external-model-64b8599cdb-z7hls\" (UID: \"e812649f-0314-4845-bc6c-8d7d4b225286\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" Oct 30 12:49:42.104010 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:42.103987 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e812649f-0314-4845-bc6c-8d7d4b225286-routing-overlay-volume\") pod \"payload-processing-external-model-64b8599cdb-z7hls\" (UID: \"e812649f-0314-4845-bc6c-8d7d4b225286\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" Oct 30 12:49:42.104108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:42.104030 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e812649f-0314-4845-bc6c-8d7d4b225286-plugins-config-volume\") pod \"payload-processing-external-model-64b8599cdb-z7hls\" (UID: \"e812649f-0314-4845-bc6c-8d7d4b225286\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" Oct 30 12:49:42.105931 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:42.105912 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e812649f-0314-4845-bc6c-8d7d4b225286-provider-credentials\") pod \"payload-processing-external-model-64b8599cdb-z7hls\" (UID: \"e812649f-0314-4845-bc6c-8d7d4b225286\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" Oct 30 12:49:42.126288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:42.126235 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7l9t\" (UniqueName: \"kubernetes.io/projected/e812649f-0314-4845-bc6c-8d7d4b225286-kube-api-access-t7l9t\") pod \"payload-processing-external-model-64b8599cdb-z7hls\" (UID: \"e812649f-0314-4845-bc6c-8d7d4b225286\") " pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" Oct 30 12:49:42.202526 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:42.202494 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" Oct 30 12:49:42.514211 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:49:42.514180 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode812649f_0314_4845_bc6c_8d7d4b225286.slice/crio-2e292dd9711e4afc6319252409a243745e8e84e5f0d6d416f118370afdd1a32c WatchSource:0}: Error finding container 2e292dd9711e4afc6319252409a243745e8e84e5f0d6d416f118370afdd1a32c: Status 404 returned error can't find the container with id 2e292dd9711e4afc6319252409a243745e8e84e5f0d6d416f118370afdd1a32c Oct 30 12:49:42.514471 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:42.514446 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls"] Oct 30 12:49:43.071775 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:43.071725 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:43.299715 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:43.299678 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" event={"ID":"e812649f-0314-4845-bc6c-8d7d4b225286","Type":"ContainerStarted","Data":"5fe96a01032becbd8950a8be1ef4c4581bcd9cfced3b31dcc1cc6f1df3375fd1"} Oct 30 12:49:43.299899 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:43.299737 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" event={"ID":"e812649f-0314-4845-bc6c-8d7d4b225286","Type":"ContainerStarted","Data":"2e292dd9711e4afc6319252409a243745e8e84e5f0d6d416f118370afdd1a32c"} Oct 30 12:49:43.300043 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:43.300026 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" Oct 30 12:49:43.343394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:43.343279 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" podStartSLOduration=2.343244645 podStartE2EDuration="2.343244645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:43.339810495 +0000 UTC m=+3336.449653376" watchObservedRunningTime="2026-10-30 12:49:43.343244645 +0000 UTC m=+3336.453087525" Oct 30 12:49:44.871133 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:44.871073 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:45.109083 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:45.109044 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" podUID="3088acb2-d284-4267-af59-7bbbef0bb012" containerName="payload-processing" containerID="cri-o://f6dee50be2e21e4944b02b5406be1a97d61bf5614627a9802b528552a01a2c74" gracePeriod=25 Oct 30 12:49:45.322182 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:45.322150 4413 generic.go:356] "Generic (PLEG): container finished" podID="3088acb2-d284-4267-af59-7bbbef0bb012" containerID="f6dee50be2e21e4944b02b5406be1a97d61bf5614627a9802b528552a01a2c74" exitCode=0 Oct 30 12:49:45.322364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:45.322230 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" event={"ID":"3088acb2-d284-4267-af59-7bbbef0bb012","Type":"ContainerDied","Data":"f6dee50be2e21e4944b02b5406be1a97d61bf5614627a9802b528552a01a2c74"} Oct 30 12:49:45.404430 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:45.404407 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" Oct 30 12:49:45.476785 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:45.476742 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3088acb2-d284-4267-af59-7bbbef0bb012-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3088acb2-d284-4267-af59-7bbbef0bb012-provider-credentials\") pod \"3088acb2-d284-4267-af59-7bbbef0bb012\" (UID: \"3088acb2-d284-4267-af59-7bbbef0bb012\") " Oct 30 12:49:45.476992 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:45.476909 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3088acb2-d284-4267-af59-7bbbef0bb012-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3088acb2-d284-4267-af59-7bbbef0bb012-plugins-config-volume\") pod \"3088acb2-d284-4267-af59-7bbbef0bb012\" (UID: \"3088acb2-d284-4267-af59-7bbbef0bb012\") " Oct 30 12:49:45.476992 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:45.476943 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3088acb2-d284-4267-af59-7bbbef0bb012-kube-api-access-6ffkl\" (UniqueName: \"kubernetes.io/projected/3088acb2-d284-4267-af59-7bbbef0bb012-kube-api-access-6ffkl\") pod \"3088acb2-d284-4267-af59-7bbbef0bb012\" (UID: \"3088acb2-d284-4267-af59-7bbbef0bb012\") " Oct 30 12:49:45.477125 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:45.477026 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3088acb2-d284-4267-af59-7bbbef0bb012-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3088acb2-d284-4267-af59-7bbbef0bb012-routing-overlay-volume\") pod \"3088acb2-d284-4267-af59-7bbbef0bb012\" (UID: \"3088acb2-d284-4267-af59-7bbbef0bb012\") " Oct 30 12:49:45.477461 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:45.477430 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3088acb2-d284-4267-af59-7bbbef0bb012-plugins-config-volume" pod "3088acb2-d284-4267-af59-7bbbef0bb012" (UID: "3088acb2-d284-4267-af59-7bbbef0bb012"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:49:45.477581 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:45.477476 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3088acb2-d284-4267-af59-7bbbef0bb012-routing-overlay-volume" pod "3088acb2-d284-4267-af59-7bbbef0bb012" (UID: "3088acb2-d284-4267-af59-7bbbef0bb012"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:49:45.478061 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:45.478040 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3088acb2-d284-4267-af59-7bbbef0bb012-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:49:45.478171 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:45.478065 4413 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3088acb2-d284-4267-af59-7bbbef0bb012-routing-overlay-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:49:45.479345 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:45.479323 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3088acb2-d284-4267-af59-7bbbef0bb012-provider-credentials" pod "3088acb2-d284-4267-af59-7bbbef0bb012" (UID: "3088acb2-d284-4267-af59-7bbbef0bb012"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:49:45.479915 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:45.479886 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3088acb2-d284-4267-af59-7bbbef0bb012-kube-api-access-6ffkl" pod "3088acb2-d284-4267-af59-7bbbef0bb012" (UID: "3088acb2-d284-4267-af59-7bbbef0bb012"). InnerVolumeSpecName "kube-api-access-6ffkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:49:45.592082 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:45.592046 4413 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3088acb2-d284-4267-af59-7bbbef0bb012-provider-credentials\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:49:45.592082 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:45.592080 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ffkl\" (UniqueName: \"kubernetes.io/projected/3088acb2-d284-4267-af59-7bbbef0bb012-kube-api-access-6ffkl\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:49:46.333018 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:46.332984 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" event={"ID":"3088acb2-d284-4267-af59-7bbbef0bb012","Type":"ContainerDied","Data":"618f7e091e5ec9131cefce637b385e9366efb9491ec27c44aa4a8113d009d10a"} Oct 30 12:49:46.333631 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:46.333054 4413 scope.go:122] "RemoveContainer" containerID="f6dee50be2e21e4944b02b5406be1a97d61bf5614627a9802b528552a01a2c74" Oct 30 12:49:46.333631 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:46.333001 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm" Oct 30 12:49:46.385689 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:46.385654 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm"] Oct 30 12:49:46.397267 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:46.397225 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-64b8599cdb-bxjhm"] Oct 30 12:49:46.571604 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:46.571539 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:47.023540 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:47.023507 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3088acb2-d284-4267-af59-7bbbef0bb012" path="/var/lib/kubelet/pods/3088acb2-d284-4267-af59-7bbbef0bb012/volumes" Oct 30 12:49:47.399768 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:47.399725 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:47.551479 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:47.551445 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:47.720629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:47.720522 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls"] Oct 30 12:49:47.724149 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:47.724122 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" Oct 30 12:49:48.557464 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:48.557432 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:49.294537 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:49.294502 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:49.435089 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:49.435056 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:50.387446 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.387413 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-696c89547b-6ftjg"] Oct 30 12:49:50.388219 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.388204 4413 cpu_manager.go:414] "containerMap: removing container" podUID="3088acb2-d284-4267-af59-7bbbef0bb012" containerName="payload-processing" Oct 30 12:49:50.388219 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.388220 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3088acb2-d284-4267-af59-7bbbef0bb012" containerName="payload-processing" Oct 30 12:49:50.388386 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.388352 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="3088acb2-d284-4267-af59-7bbbef0bb012" containerName="payload-processing" Oct 30 12:49:50.393547 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.393510 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" Oct 30 12:49:50.410818 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.410787 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-696c89547b-6ftjg"] Oct 30 12:49:50.418442 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.418419 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9d0d04e9097f\"" Oct 30 12:49:50.470410 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.470378 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85a4061f-ba8a-497f-b338-c239f666aa9c-provider-credentials\") pod \"payload-processing-external-model-696c89547b-6ftjg\" (UID: \"85a4061f-ba8a-497f-b338-c239f666aa9c\") " pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" Oct 30 12:49:50.470570 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.470443 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66c2h\" (UniqueName: \"kubernetes.io/projected/85a4061f-ba8a-497f-b338-c239f666aa9c-kube-api-access-66c2h\") pod \"payload-processing-external-model-696c89547b-6ftjg\" (UID: \"85a4061f-ba8a-497f-b338-c239f666aa9c\") " pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" Oct 30 12:49:50.470632 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.470607 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85a4061f-ba8a-497f-b338-c239f666aa9c-routing-overlay-volume\") pod \"payload-processing-external-model-696c89547b-6ftjg\" (UID: \"85a4061f-ba8a-497f-b338-c239f666aa9c\") " pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" Oct 30 12:49:50.470788 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.470776 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a4061f-ba8a-497f-b338-c239f666aa9c-plugins-config-volume\") pod \"payload-processing-external-model-696c89547b-6ftjg\" (UID: \"85a4061f-ba8a-497f-b338-c239f666aa9c\") " pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" Oct 30 12:49:50.473029 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.473005 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:50.572473 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.572442 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85a4061f-ba8a-497f-b338-c239f666aa9c-routing-overlay-volume\") pod \"payload-processing-external-model-696c89547b-6ftjg\" (UID: \"85a4061f-ba8a-497f-b338-c239f666aa9c\") " pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" Oct 30 12:49:50.572667 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.572585 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a4061f-ba8a-497f-b338-c239f666aa9c-plugins-config-volume\") pod \"payload-processing-external-model-696c89547b-6ftjg\" (UID: \"85a4061f-ba8a-497f-b338-c239f666aa9c\") " pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" Oct 30 12:49:50.572736 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.572687 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85a4061f-ba8a-497f-b338-c239f666aa9c-provider-credentials\") pod \"payload-processing-external-model-696c89547b-6ftjg\" (UID: \"85a4061f-ba8a-497f-b338-c239f666aa9c\") " pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" Oct 30 12:49:50.572797 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.572737 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66c2h\" (UniqueName: \"kubernetes.io/projected/85a4061f-ba8a-497f-b338-c239f666aa9c-kube-api-access-66c2h\") pod \"payload-processing-external-model-696c89547b-6ftjg\" (UID: \"85a4061f-ba8a-497f-b338-c239f666aa9c\") " pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" Oct 30 12:49:50.573211 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.573185 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85a4061f-ba8a-497f-b338-c239f666aa9c-routing-overlay-volume\") pod \"payload-processing-external-model-696c89547b-6ftjg\" (UID: \"85a4061f-ba8a-497f-b338-c239f666aa9c\") " pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" Oct 30 12:49:50.573382 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.573362 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a4061f-ba8a-497f-b338-c239f666aa9c-plugins-config-volume\") pod \"payload-processing-external-model-696c89547b-6ftjg\" (UID: \"85a4061f-ba8a-497f-b338-c239f666aa9c\") " pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" Oct 30 12:49:50.575584 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.575562 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85a4061f-ba8a-497f-b338-c239f666aa9c-provider-credentials\") pod \"payload-processing-external-model-696c89547b-6ftjg\" (UID: \"85a4061f-ba8a-497f-b338-c239f666aa9c\") " pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" Oct 30 12:49:50.596206 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.596176 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66c2h\" (UniqueName: \"kubernetes.io/projected/85a4061f-ba8a-497f-b338-c239f666aa9c-kube-api-access-66c2h\") pod \"payload-processing-external-model-696c89547b-6ftjg\" (UID: \"85a4061f-ba8a-497f-b338-c239f666aa9c\") " pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" Oct 30 12:49:50.709469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:50.709389 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" Oct 30 12:49:51.017725 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:49:51.017693 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a4061f_ba8a_497f_b338_c239f666aa9c.slice/crio-22cb027c5f8cbec98d74a5eb42a47be8a767866cd4499785e5875e32d0d09286 WatchSource:0}: Error finding container 22cb027c5f8cbec98d74a5eb42a47be8a767866cd4499785e5875e32d0d09286: Status 404 returned error can't find the container with id 22cb027c5f8cbec98d74a5eb42a47be8a767866cd4499785e5875e32d0d09286 Oct 30 12:49:51.027062 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:51.027035 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-696c89547b-6ftjg"] Oct 30 12:49:51.380858 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:51.380826 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" event={"ID":"85a4061f-ba8a-497f-b338-c239f666aa9c","Type":"ContainerStarted","Data":"86f7d5f6bc03f828c911a4efd5ee58f47d77def3099b32692befc33a207f6924"} Oct 30 12:49:51.381035 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:51.380874 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" event={"ID":"85a4061f-ba8a-497f-b338-c239f666aa9c","Type":"ContainerStarted","Data":"22cb027c5f8cbec98d74a5eb42a47be8a767866cd4499785e5875e32d0d09286"} Oct 30 12:49:51.381035 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:51.380905 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" Oct 30 12:49:51.432658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:51.432610 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" podStartSLOduration=1.432594882 podStartE2EDuration="1.432594882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:49:51.432061664 +0000 UTC m=+3344.541904545" watchObservedRunningTime="2026-10-30 12:49:51.432594882 +0000 UTC m=+3344.542437762" Oct 30 12:49:52.181871 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:52.181834 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:52.723417 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:52.723259 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" podUID="e812649f-0314-4845-bc6c-8d7d4b225286" containerName="payload-processing" containerID="cri-o://5fe96a01032becbd8950a8be1ef4c4581bcd9cfced3b31dcc1cc6f1df3375fd1" gracePeriod=25 Oct 30 12:49:53.008323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.008294 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" Oct 30 12:49:53.112322 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.112284 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e812649f-0314-4845-bc6c-8d7d4b225286-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e812649f-0314-4845-bc6c-8d7d4b225286-provider-credentials\") pod \"e812649f-0314-4845-bc6c-8d7d4b225286\" (UID: \"e812649f-0314-4845-bc6c-8d7d4b225286\") " Oct 30 12:49:53.112518 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.112402 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e812649f-0314-4845-bc6c-8d7d4b225286-kube-api-access-t7l9t\" (UniqueName: \"kubernetes.io/projected/e812649f-0314-4845-bc6c-8d7d4b225286-kube-api-access-t7l9t\") pod \"e812649f-0314-4845-bc6c-8d7d4b225286\" (UID: \"e812649f-0314-4845-bc6c-8d7d4b225286\") " Oct 30 12:49:53.112518 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.112507 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e812649f-0314-4845-bc6c-8d7d4b225286-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e812649f-0314-4845-bc6c-8d7d4b225286-routing-overlay-volume\") pod \"e812649f-0314-4845-bc6c-8d7d4b225286\" (UID: \"e812649f-0314-4845-bc6c-8d7d4b225286\") " Oct 30 12:49:53.112655 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.112568 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e812649f-0314-4845-bc6c-8d7d4b225286-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e812649f-0314-4845-bc6c-8d7d4b225286-plugins-config-volume\") pod \"e812649f-0314-4845-bc6c-8d7d4b225286\" (UID: \"e812649f-0314-4845-bc6c-8d7d4b225286\") " Oct 30 12:49:53.112910 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.112886 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e812649f-0314-4845-bc6c-8d7d4b225286-routing-overlay-volume" pod "e812649f-0314-4845-bc6c-8d7d4b225286" (UID: "e812649f-0314-4845-bc6c-8d7d4b225286"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:49:53.113497 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.112997 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e812649f-0314-4845-bc6c-8d7d4b225286-plugins-config-volume" pod "e812649f-0314-4845-bc6c-8d7d4b225286" (UID: "e812649f-0314-4845-bc6c-8d7d4b225286"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:49:53.114888 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.114855 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e812649f-0314-4845-bc6c-8d7d4b225286-kube-api-access-t7l9t" pod "e812649f-0314-4845-bc6c-8d7d4b225286" (UID: "e812649f-0314-4845-bc6c-8d7d4b225286"). InnerVolumeSpecName "kube-api-access-t7l9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:49:53.115044 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.115024 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e812649f-0314-4845-bc6c-8d7d4b225286-provider-credentials" pod "e812649f-0314-4845-bc6c-8d7d4b225286" (UID: "e812649f-0314-4845-bc6c-8d7d4b225286"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:49:53.214906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.214873 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7l9t\" (UniqueName: \"kubernetes.io/projected/e812649f-0314-4845-bc6c-8d7d4b225286-kube-api-access-t7l9t\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:49:53.214906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.214900 4413 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e812649f-0314-4845-bc6c-8d7d4b225286-routing-overlay-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:49:53.214906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.214910 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e812649f-0314-4845-bc6c-8d7d4b225286-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:49:53.215156 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.214918 4413 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e812649f-0314-4845-bc6c-8d7d4b225286-provider-credentials\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:49:53.398112 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.398077 4413 generic.go:356] "Generic (PLEG): container finished" podID="e812649f-0314-4845-bc6c-8d7d4b225286" containerID="5fe96a01032becbd8950a8be1ef4c4581bcd9cfced3b31dcc1cc6f1df3375fd1" exitCode=0 Oct 30 12:49:53.398315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.398131 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" Oct 30 12:49:53.398315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.398136 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" event={"ID":"e812649f-0314-4845-bc6c-8d7d4b225286","Type":"ContainerDied","Data":"5fe96a01032becbd8950a8be1ef4c4581bcd9cfced3b31dcc1cc6f1df3375fd1"} Oct 30 12:49:53.398315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.398192 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls" event={"ID":"e812649f-0314-4845-bc6c-8d7d4b225286","Type":"ContainerDied","Data":"2e292dd9711e4afc6319252409a243745e8e84e5f0d6d416f118370afdd1a32c"} Oct 30 12:49:53.398315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.398221 4413 scope.go:122] "RemoveContainer" containerID="5fe96a01032becbd8950a8be1ef4c4581bcd9cfced3b31dcc1cc6f1df3375fd1" Oct 30 12:49:53.409839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.409815 4413 scope.go:122] "RemoveContainer" containerID="5fe96a01032becbd8950a8be1ef4c4581bcd9cfced3b31dcc1cc6f1df3375fd1" Oct 30 12:49:53.410147 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:49:53.410129 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fe96a01032becbd8950a8be1ef4c4581bcd9cfced3b31dcc1cc6f1df3375fd1\": container with ID starting with 5fe96a01032becbd8950a8be1ef4c4581bcd9cfced3b31dcc1cc6f1df3375fd1 not found: ID does not exist" containerID="5fe96a01032becbd8950a8be1ef4c4581bcd9cfced3b31dcc1cc6f1df3375fd1" Oct 30 12:49:53.410200 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.410158 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe96a01032becbd8950a8be1ef4c4581bcd9cfced3b31dcc1cc6f1df3375fd1"} err="failed to get container status \"5fe96a01032becbd8950a8be1ef4c4581bcd9cfced3b31dcc1cc6f1df3375fd1\": rpc error: code = NotFound desc = could not find container \"5fe96a01032becbd8950a8be1ef4c4581bcd9cfced3b31dcc1cc6f1df3375fd1\": container with ID starting with 5fe96a01032becbd8950a8be1ef4c4581bcd9cfced3b31dcc1cc6f1df3375fd1 not found: ID does not exist" Oct 30 12:49:53.450013 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.449982 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls"] Oct 30 12:49:53.462106 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.462078 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-64b8599cdb-z7hls"] Oct 30 12:49:53.877636 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:53.877596 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:55.022605 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:55.022569 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e812649f-0314-4845-bc6c-8d7d4b225286" path="/var/lib/kubelet/pods/e812649f-0314-4845-bc6c-8d7d4b225286/volumes" Oct 30 12:49:55.539424 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:55.539392 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:55.773931 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:55.773904 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:55.895348 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:55.895313 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:56.091865 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:56.091834 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-696c89547b-6ftjg"] Oct 30 12:49:56.094623 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:56.094604 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" Oct 30 12:49:57.269041 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:57.269008 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:58.592602 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:58.592554 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:58.764228 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:58.764183 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:58.885525 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:58.885494 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:59.005136 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:59.005100 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:59.129970 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:59.129940 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:59.649134 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:59.649096 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:49:59.790235 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:49:59.790205 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:00.907512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:00.907484 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:01.030381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.030351 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:01.092998 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.092962 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" podUID="85a4061f-ba8a-497f-b338-c239f666aa9c" containerName="payload-processing" containerID="cri-o://86f7d5f6bc03f828c911a4efd5ee58f47d77def3099b32692befc33a207f6924" gracePeriod=25 Oct 30 12:50:01.139824 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.139797 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:01.345849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.345809 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk"] Oct 30 12:50:01.347041 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.347019 4413 cpu_manager.go:414] "containerMap: removing container" podUID="e812649f-0314-4845-bc6c-8d7d4b225286" containerName="payload-processing" Oct 30 12:50:01.347099 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.347046 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e812649f-0314-4845-bc6c-8d7d4b225286" containerName="payload-processing" Oct 30 12:50:01.347352 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.347337 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="e812649f-0314-4845-bc6c-8d7d4b225286" containerName="payload-processing" Oct 30 12:50:01.353299 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.353276 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" Oct 30 12:50:01.377975 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.377947 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk"] Oct 30 12:50:01.378861 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.378842 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12fd33\"/\"routing-overlay\"" Oct 30 12:50:01.379022 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.378842 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12fd33\"/\"openshift-service-ca.crt\"" Oct 30 12:50:01.379022 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.378891 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12fd33\"/\"kube-root-ca.crt\"" Oct 30 12:50:01.379318 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.378961 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12fd33\"/\"payload-processing-external-model-plugins-e2e-extmod-b-12fd33\"" Oct 30 12:50:01.379318 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.379021 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12fd33\"/\"agc-cred-c49248e8d354\"" Oct 30 12:50:01.380414 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.380396 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12fd33\"/\"payload-processing-external-model-e2e--54fdd6e5-dockercfg-6pxch\"" Oct 30 12:50:01.429388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.429364 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" Oct 30 12:50:01.450725 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.450697 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a4061f-ba8a-497f-b338-c239f666aa9c-plugins-config-volume" pod "85a4061f-ba8a-497f-b338-c239f666aa9c" (UID: "85a4061f-ba8a-497f-b338-c239f666aa9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:01.450924 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.450238 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85a4061f-ba8a-497f-b338-c239f666aa9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a4061f-ba8a-497f-b338-c239f666aa9c-plugins-config-volume\") pod \"85a4061f-ba8a-497f-b338-c239f666aa9c\" (UID: \"85a4061f-ba8a-497f-b338-c239f666aa9c\") " Oct 30 12:50:01.451240 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.451221 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85a4061f-ba8a-497f-b338-c239f666aa9c-kube-api-access-66c2h\" (UniqueName: \"kubernetes.io/projected/85a4061f-ba8a-497f-b338-c239f666aa9c-kube-api-access-66c2h\") pod \"85a4061f-ba8a-497f-b338-c239f666aa9c\" (UID: \"85a4061f-ba8a-497f-b338-c239f666aa9c\") " Oct 30 12:50:01.451784 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.451765 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85a4061f-ba8a-497f-b338-c239f666aa9c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85a4061f-ba8a-497f-b338-c239f666aa9c-routing-overlay-volume\") pod \"85a4061f-ba8a-497f-b338-c239f666aa9c\" (UID: \"85a4061f-ba8a-497f-b338-c239f666aa9c\") " Oct 30 12:50:01.451992 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.451952 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85a4061f-ba8a-497f-b338-c239f666aa9c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/85a4061f-ba8a-497f-b338-c239f666aa9c-provider-credentials\") pod \"85a4061f-ba8a-497f-b338-c239f666aa9c\" (UID: \"85a4061f-ba8a-497f-b338-c239f666aa9c\") " Oct 30 12:50:01.452621 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.452174 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a4061f-ba8a-497f-b338-c239f666aa9c-routing-overlay-volume" pod "85a4061f-ba8a-497f-b338-c239f666aa9c" (UID: "85a4061f-ba8a-497f-b338-c239f666aa9c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:50:01.452892 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.452869 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdcdk\" (UniqueName: \"kubernetes.io/projected/22fd6023-a535-4f01-ace9-593c8381536e-kube-api-access-wdcdk\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk\" (UID: \"22fd6023-a535-4f01-ace9-593c8381536e\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" Oct 30 12:50:01.453226 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.453209 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22fd6023-a535-4f01-ace9-593c8381536e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk\" (UID: \"22fd6023-a535-4f01-ace9-593c8381536e\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" Oct 30 12:50:01.453477 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.453459 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fd6023-a535-4f01-ace9-593c8381536e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk\" (UID: \"22fd6023-a535-4f01-ace9-593c8381536e\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" Oct 30 12:50:01.453710 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.453691 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22fd6023-a535-4f01-ace9-593c8381536e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk\" (UID: \"22fd6023-a535-4f01-ace9-593c8381536e\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" Oct 30 12:50:01.453956 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.453939 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a4061f-ba8a-497f-b338-c239f666aa9c-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:50:01.454061 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.454049 4413 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85a4061f-ba8a-497f-b338-c239f666aa9c-routing-overlay-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:50:01.458126 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.456389 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a4061f-ba8a-497f-b338-c239f666aa9c-kube-api-access-66c2h" pod "85a4061f-ba8a-497f-b338-c239f666aa9c" (UID: "85a4061f-ba8a-497f-b338-c239f666aa9c"). InnerVolumeSpecName "kube-api-access-66c2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:01.458784 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.458748 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a4061f-ba8a-497f-b338-c239f666aa9c-provider-credentials" pod "85a4061f-ba8a-497f-b338-c239f666aa9c" (UID: "85a4061f-ba8a-497f-b338-c239f666aa9c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:50:01.568597 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.561044 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22fd6023-a535-4f01-ace9-593c8381536e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk\" (UID: \"22fd6023-a535-4f01-ace9-593c8381536e\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" Oct 30 12:50:01.568597 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.561136 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fd6023-a535-4f01-ace9-593c8381536e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk\" (UID: \"22fd6023-a535-4f01-ace9-593c8381536e\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" Oct 30 12:50:01.568597 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.561188 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22fd6023-a535-4f01-ace9-593c8381536e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk\" (UID: \"22fd6023-a535-4f01-ace9-593c8381536e\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" Oct 30 12:50:01.568597 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.561326 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdcdk\" (UniqueName: \"kubernetes.io/projected/22fd6023-a535-4f01-ace9-593c8381536e-kube-api-access-wdcdk\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk\" (UID: \"22fd6023-a535-4f01-ace9-593c8381536e\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" Oct 30 12:50:01.568597 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.561533 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66c2h\" (UniqueName: \"kubernetes.io/projected/85a4061f-ba8a-497f-b338-c239f666aa9c-kube-api-access-66c2h\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:50:01.568597 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.561548 4413 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85a4061f-ba8a-497f-b338-c239f666aa9c-provider-credentials\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:50:01.568597 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.562628 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22fd6023-a535-4f01-ace9-593c8381536e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk\" (UID: \"22fd6023-a535-4f01-ace9-593c8381536e\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" Oct 30 12:50:01.568597 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.563372 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fd6023-a535-4f01-ace9-593c8381536e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk\" (UID: \"22fd6023-a535-4f01-ace9-593c8381536e\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" Oct 30 12:50:01.568597 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.566540 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22fd6023-a535-4f01-ace9-593c8381536e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk\" (UID: \"22fd6023-a535-4f01-ace9-593c8381536e\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" Oct 30 12:50:01.575321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.572767 4413 generic.go:356] "Generic (PLEG): container finished" podID="85a4061f-ba8a-497f-b338-c239f666aa9c" containerID="86f7d5f6bc03f828c911a4efd5ee58f47d77def3099b32692befc33a207f6924" exitCode=0 Oct 30 12:50:01.575321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.573016 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" event={"ID":"85a4061f-ba8a-497f-b338-c239f666aa9c","Type":"ContainerDied","Data":"86f7d5f6bc03f828c911a4efd5ee58f47d77def3099b32692befc33a207f6924"} Oct 30 12:50:01.575321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.573065 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" event={"ID":"85a4061f-ba8a-497f-b338-c239f666aa9c","Type":"ContainerDied","Data":"22cb027c5f8cbec98d74a5eb42a47be8a767866cd4499785e5875e32d0d09286"} Oct 30 12:50:01.575321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.573095 4413 scope.go:122] "RemoveContainer" containerID="86f7d5f6bc03f828c911a4efd5ee58f47d77def3099b32692befc33a207f6924" Oct 30 12:50:01.575321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.573353 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-696c89547b-6ftjg" Oct 30 12:50:01.601281 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.600144 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdcdk\" (UniqueName: \"kubernetes.io/projected/22fd6023-a535-4f01-ace9-593c8381536e-kube-api-access-wdcdk\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk\" (UID: \"22fd6023-a535-4f01-ace9-593c8381536e\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" Oct 30 12:50:01.604784 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.604762 4413 scope.go:122] "RemoveContainer" containerID="86f7d5f6bc03f828c911a4efd5ee58f47d77def3099b32692befc33a207f6924" Oct 30 12:50:01.605228 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:50:01.605202 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f7d5f6bc03f828c911a4efd5ee58f47d77def3099b32692befc33a207f6924\": container with ID starting with 86f7d5f6bc03f828c911a4efd5ee58f47d77def3099b32692befc33a207f6924 not found: ID does not exist" containerID="86f7d5f6bc03f828c911a4efd5ee58f47d77def3099b32692befc33a207f6924" Oct 30 12:50:01.605342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.605241 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f7d5f6bc03f828c911a4efd5ee58f47d77def3099b32692befc33a207f6924"} err="failed to get container status \"86f7d5f6bc03f828c911a4efd5ee58f47d77def3099b32692befc33a207f6924\": rpc error: code = NotFound desc = could not find container \"86f7d5f6bc03f828c911a4efd5ee58f47d77def3099b32692befc33a207f6924\": container with ID starting with 86f7d5f6bc03f828c911a4efd5ee58f47d77def3099b32692befc33a207f6924 not found: ID does not exist" Oct 30 12:50:01.636694 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.636662 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-696c89547b-6ftjg"] Oct 30 12:50:01.640380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.640355 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-696c89547b-6ftjg"] Oct 30 12:50:01.684636 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:01.684592 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" Oct 30 12:50:02.002395 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.002363 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk"] Oct 30 12:50:02.003058 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:50:02.003031 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fd6023_a535_4f01_ace9_593c8381536e.slice/crio-bdb16a853292ef941d0e8f51f6386d7097e8c14302f1351c1557162dbfb2dad2 WatchSource:0}: Error finding container bdb16a853292ef941d0e8f51f6386d7097e8c14302f1351c1557162dbfb2dad2: Status 404 returned error can't find the container with id bdb16a853292ef941d0e8f51f6386d7097e8c14302f1351c1557162dbfb2dad2 Oct 30 12:50:02.052139 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.052114 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:02.312420 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.312348 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw"] Oct 30 12:50:02.313169 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.313150 4413 cpu_manager.go:414] "containerMap: removing container" podUID="85a4061f-ba8a-497f-b338-c239f666aa9c" containerName="payload-processing" Oct 30 12:50:02.313169 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.313168 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85a4061f-ba8a-497f-b338-c239f666aa9c" containerName="payload-processing" Oct 30 12:50:02.313384 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.313310 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="85a4061f-ba8a-497f-b338-c239f666aa9c" containerName="payload-processing" Oct 30 12:50:02.317170 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.317152 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" Oct 30 12:50:02.334200 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.334177 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"agc-cred-bbd00a81253a\"" Oct 30 12:50:02.334376 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.334216 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"routing-overlay\"" Oct 30 12:50:02.335180 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.335158 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw"] Oct 30 12:50:02.342632 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.342614 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"kube-root-ca.crt\"" Oct 30 12:50:02.342804 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.342638 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"payload-processing-external-model-e2e--03fcd75d-dockercfg-5q2hg\"" Oct 30 12:50:02.342804 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.342678 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"openshift-service-ca.crt\"" Oct 30 12:50:02.342995 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.342709 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2545c7\"" Oct 30 12:50:02.373535 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.373509 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbzhp\" (UniqueName: \"kubernetes.io/projected/41e088cd-2edf-4b23-bea8-26dc6a2318ab-kube-api-access-rbzhp\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw\" (UID: \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" Oct 30 12:50:02.373690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.373676 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41e088cd-2edf-4b23-bea8-26dc6a2318ab-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw\" (UID: \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" Oct 30 12:50:02.373792 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.373773 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e088cd-2edf-4b23-bea8-26dc6a2318ab-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw\" (UID: \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" Oct 30 12:50:02.373835 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.373822 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41e088cd-2edf-4b23-bea8-26dc6a2318ab-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw\" (UID: \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" Oct 30 12:50:02.475090 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.475044 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41e088cd-2edf-4b23-bea8-26dc6a2318ab-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw\" (UID: \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" Oct 30 12:50:02.475357 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.475282 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbzhp\" (UniqueName: \"kubernetes.io/projected/41e088cd-2edf-4b23-bea8-26dc6a2318ab-kube-api-access-rbzhp\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw\" (UID: \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" Oct 30 12:50:02.475460 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.475441 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41e088cd-2edf-4b23-bea8-26dc6a2318ab-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw\" (UID: \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" Oct 30 12:50:02.475543 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.475515 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e088cd-2edf-4b23-bea8-26dc6a2318ab-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw\" (UID: \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" Oct 30 12:50:02.476283 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.476220 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e088cd-2edf-4b23-bea8-26dc6a2318ab-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw\" (UID: \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" Oct 30 12:50:02.486366 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.483329 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41e088cd-2edf-4b23-bea8-26dc6a2318ab-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw\" (UID: \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" Oct 30 12:50:02.486569 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.486547 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41e088cd-2edf-4b23-bea8-26dc6a2318ab-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw\" (UID: \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" Oct 30 12:50:02.510367 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.507297 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbzhp\" (UniqueName: \"kubernetes.io/projected/41e088cd-2edf-4b23-bea8-26dc6a2318ab-kube-api-access-rbzhp\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw\" (UID: \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" Oct 30 12:50:02.581988 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.581897 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" event={"ID":"22fd6023-a535-4f01-ace9-593c8381536e","Type":"ContainerStarted","Data":"47ed23327b4473bbd19ad76d7806ff334fa317e39c6769d8207f5db22faf934b"} Oct 30 12:50:02.581988 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.581960 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" event={"ID":"22fd6023-a535-4f01-ace9-593c8381536e","Type":"ContainerStarted","Data":"bdb16a853292ef941d0e8f51f6386d7097e8c14302f1351c1557162dbfb2dad2"} Oct 30 12:50:02.581988 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.581988 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" Oct 30 12:50:02.627834 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.627788 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" podStartSLOduration=1.6277741350000001 podStartE2EDuration="1.627774135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:02.623992896 +0000 UTC m=+3355.733835795" watchObservedRunningTime="2026-10-30 12:50:02.627774135 +0000 UTC m=+3355.737617059" Oct 30 12:50:02.632997 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.632973 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" Oct 30 12:50:02.754310 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.754205 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:02.940347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:02.940314 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw"] Oct 30 12:50:02.940935 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:50:02.940911 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e088cd_2edf_4b23_bea8_26dc6a2318ab.slice/crio-6b14268dd9a608dde00578c56f541a6f22c6978b95b0235f56dd861e4759aa22 WatchSource:0}: Error finding container 6b14268dd9a608dde00578c56f541a6f22c6978b95b0235f56dd861e4759aa22: Status 404 returned error can't find the container with id 6b14268dd9a608dde00578c56f541a6f22c6978b95b0235f56dd861e4759aa22 Oct 30 12:50:03.022633 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:03.022608 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85a4061f-ba8a-497f-b338-c239f666aa9c" path="/var/lib/kubelet/pods/85a4061f-ba8a-497f-b338-c239f666aa9c/volumes" Oct 30 12:50:03.592282 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:03.592225 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" event={"ID":"41e088cd-2edf-4b23-bea8-26dc6a2318ab","Type":"ContainerStarted","Data":"a6018b01c00f969e622d2ae45b9653d68368915fea281494980274454a4452e7"} Oct 30 12:50:03.592486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:03.592300 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" event={"ID":"41e088cd-2edf-4b23-bea8-26dc6a2318ab","Type":"ContainerStarted","Data":"6b14268dd9a608dde00578c56f541a6f22c6978b95b0235f56dd861e4759aa22"} Oct 30 12:50:03.592486 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:03.592327 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" Oct 30 12:50:03.635760 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:03.635709 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" podStartSLOduration=1.635695865 podStartE2EDuration="1.635695865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:03.633063984 +0000 UTC m=+3356.742906865" watchObservedRunningTime="2026-10-30 12:50:03.635695865 +0000 UTC m=+3356.745538745" Oct 30 12:50:04.357003 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:04.356960 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:05.816960 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:50:05.816922 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:50:06.475018 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:06.474958 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:06.984386 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:06.984338 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:07.154277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:07.153238 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:07.270217 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:07.270056 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:08.712567 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:08.696939 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:09.244520 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:09.244485 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:09.359676 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:09.359627 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:10.365757 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:10.365717 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:10.491497 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:10.491442 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:10.652872 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:10.652832 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:12.457511 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:12.457475 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:13.596446 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:13.596412 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" Oct 30 12:50:14.134030 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:14.133982 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:14.604232 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:14.604152 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" Oct 30 12:50:15.723959 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:15.723900 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:17.384741 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:17.384704 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:19.043002 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:19.042888 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:20.630521 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:20.630492 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:22.299307 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:22.299270 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:22.429572 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:22.429532 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:22.556860 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:22.556745 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:22.684182 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:22.684152 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:23.506419 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:23.506385 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:23.632361 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:23.632323 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:24.138793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:24.138759 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:24.328445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:24.328411 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:24.615001 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:24.614928 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:26.072293 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:26.072244 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:27.947597 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:27.947549 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:28.668951 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:28.668901 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:29.305397 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:29.305354 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:50:29.306046 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:29.306009 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" podUID="fe31c94d-c481-44ee-8183-296ce6090146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:50:29.969698 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:29.969577 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:31.355777 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:31.355731 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:31.471387 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:31.471356 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:31.607600 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:31.607519 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:31.869450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:31.869370 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:33.476770 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:33.476716 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:35.071324 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:35.071285 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:35.815619 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:50:35.815578 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:50:36.784484 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:36.784451 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:37.999916 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:37.999882 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:38.119505 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:38.119469 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:38.541644 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:38.541581 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:38.978562 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:38.978491 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:39.140074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:39.140034 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:40.556659 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:40.556595 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:42.282845 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:42.282818 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:44.031106 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:44.031065 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:45.768558 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:45.768524 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:47.467027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:47.466981 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:49.240314 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:49.240274 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:50.810169 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:50.810116 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:52.502558 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:52.502526 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:52.645470 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:52.645436 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:52.759518 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:52.759429 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:53.662612 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:53.662564 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:53.775199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:53.775163 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:54.248232 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:54.248203 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:54.636114 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:54.636081 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:56.487860 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:56.487824 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:58.250383 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:58.250323 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:50:59.985180 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:50:59.985147 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:01.028994 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:01.028961 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:01.213601 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:01.213571 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:01.413144 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:01.413114 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:01.816767 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:01.816679 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:03.617050 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:03.617014 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:05.172469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:05.172440 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:05.813970 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:51:05.813933 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:51:06.860918 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:06.860857 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:08.579994 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:08.579958 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:08.890864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:08.890817 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:09.129572 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:09.129525 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:09.391743 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:09.391703 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:09.534741 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:09.534704 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:09.674906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:09.674815 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:11.062282 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:11.062225 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:13.101072 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:13.101036 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:14.797474 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:14.797437 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:15.817555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:15.817514 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:15.954897 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:15.954860 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:16.933792 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:16.933743 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:17.342828 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:17.342739 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:18.289474 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:18.289443 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:18.812568 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:18.812530 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:20.495445 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:20.495412 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:22.205095 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:22.205063 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:23.965021 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:23.964979 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:25.684277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:25.684232 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:27.546496 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:27.546462 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:29.310497 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:29.310465 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:31.014120 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:31.014088 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:32.996542 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:32.996496 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:34.943625 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:34.943572 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:35.815760 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:51:35.815719 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:51:36.482013 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:36.481979 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:38.293656 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:38.293609 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:39.961700 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:39.961666 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:41.852173 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:41.852136 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:43.579960 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:43.579911 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:45.349140 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:45.349106 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:47.087406 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:47.087370 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:48.891543 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:48.891499 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:50.556229 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:50.556173 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:52.168286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:52.168238 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:53.174464 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:53.174433 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx"] Oct 30 12:51:53.174935 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:53.174639 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" podUID="6d871b22-0655-4cc4-9afb-686f9429693f" containerName="istio-proxy" containerID="cri-o://0194586883c3a81b4ec3633f51fca293ab37e9e6ded84c7441425550cbe1d0fe" gracePeriod=30 Oct 30 12:51:53.958617 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:53.958573 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:55.650710 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:55.650674 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:56.278046 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:56.278012 4413 patch_prober.go:29] interesting pod/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 12:51:56.278232 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:56.278090 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" podUID="6d871b22-0655-4cc4-9afb-686f9429693f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 12:51:57.313205 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:57.313167 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:58.021321 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.021282 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:58.136307 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.136276 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:58.460050 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.460022 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:51:58.604325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.604198 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-workload-socket\") pod \"6d871b22-0655-4cc4-9afb-686f9429693f\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " Oct 30 12:51:58.604325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.604244 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d871b22-0655-4cc4-9afb-686f9429693f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d871b22-0655-4cc4-9afb-686f9429693f-istiod-ca-cert\") pod \"6d871b22-0655-4cc4-9afb-686f9429693f\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " Oct 30 12:51:58.604325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.604317 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d871b22-0655-4cc4-9afb-686f9429693f-kube-api-access-k97td\" (UniqueName: \"kubernetes.io/projected/6d871b22-0655-4cc4-9afb-686f9429693f-kube-api-access-k97td\") pod \"6d871b22-0655-4cc4-9afb-686f9429693f\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " Oct 30 12:51:58.604641 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.604364 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-workload-certs\") pod \"6d871b22-0655-4cc4-9afb-686f9429693f\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " Oct 30 12:51:58.604641 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.604430 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d871b22-0655-4cc4-9afb-686f9429693f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d871b22-0655-4cc4-9afb-686f9429693f-istio-podinfo\") pod \"6d871b22-0655-4cc4-9afb-686f9429693f\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " Oct 30 12:51:58.604641 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.604486 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d871b22-0655-4cc4-9afb-686f9429693f-istio-token\" (UniqueName: \"kubernetes.io/projected/6d871b22-0655-4cc4-9afb-686f9429693f-istio-token\") pod \"6d871b22-0655-4cc4-9afb-686f9429693f\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " Oct 30 12:51:58.604641 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.604515 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-istio-data\") pod \"6d871b22-0655-4cc4-9afb-686f9429693f\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " Oct 30 12:51:58.604641 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.604545 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-workload-socket" pod "6d871b22-0655-4cc4-9afb-686f9429693f" (UID: "6d871b22-0655-4cc4-9afb-686f9429693f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:51:58.604641 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.604561 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-credential-socket\") pod \"6d871b22-0655-4cc4-9afb-686f9429693f\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " Oct 30 12:51:58.605037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.604680 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d871b22-0655-4cc4-9afb-686f9429693f-istiod-ca-cert" pod "6d871b22-0655-4cc4-9afb-686f9429693f" (UID: "6d871b22-0655-4cc4-9afb-686f9429693f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:51:58.605037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.604734 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-workload-certs" pod "6d871b22-0655-4cc4-9afb-686f9429693f" (UID: "6d871b22-0655-4cc4-9afb-686f9429693f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:51:58.605037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.604763 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-istio-envoy\") pod \"6d871b22-0655-4cc4-9afb-686f9429693f\" (UID: \"6d871b22-0655-4cc4-9afb-686f9429693f\") " Oct 30 12:51:58.605037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.604902 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-credential-socket" pod "6d871b22-0655-4cc4-9afb-686f9429693f" (UID: "6d871b22-0655-4cc4-9afb-686f9429693f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:51:58.605325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.605300 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-istio-data" pod "6d871b22-0655-4cc4-9afb-686f9429693f" (UID: "6d871b22-0655-4cc4-9afb-686f9429693f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:51:58.605416 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.605359 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:51:58.605416 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.605375 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:51:58.605416 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.605384 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d871b22-0655-4cc4-9afb-686f9429693f-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:51:58.605416 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.605392 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:51:58.606914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.606883 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d871b22-0655-4cc4-9afb-686f9429693f-istio-podinfo" pod "6d871b22-0655-4cc4-9afb-686f9429693f" (UID: "6d871b22-0655-4cc4-9afb-686f9429693f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 12:51:58.607277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.607236 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-istio-envoy" pod "6d871b22-0655-4cc4-9afb-686f9429693f" (UID: "6d871b22-0655-4cc4-9afb-686f9429693f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 12:51:58.607277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.607243 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d871b22-0655-4cc4-9afb-686f9429693f-kube-api-access-k97td" pod "6d871b22-0655-4cc4-9afb-686f9429693f" (UID: "6d871b22-0655-4cc4-9afb-686f9429693f"). InnerVolumeSpecName "kube-api-access-k97td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:51:58.607405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.607308 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d871b22-0655-4cc4-9afb-686f9429693f-istio-token" pod "6d871b22-0655-4cc4-9afb-686f9429693f" (UID: "6d871b22-0655-4cc4-9afb-686f9429693f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:51:58.627262 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.627223 4413 generic.go:356] "Generic (PLEG): container finished" podID="6d871b22-0655-4cc4-9afb-686f9429693f" containerID="0194586883c3a81b4ec3633f51fca293ab37e9e6ded84c7441425550cbe1d0fe" exitCode=0 Oct 30 12:51:58.627363 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.627283 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" Oct 30 12:51:58.627363 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.627297 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" event={"ID":"6d871b22-0655-4cc4-9afb-686f9429693f","Type":"ContainerDied","Data":"0194586883c3a81b4ec3633f51fca293ab37e9e6ded84c7441425550cbe1d0fe"} Oct 30 12:51:58.627438 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.627372 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx" event={"ID":"6d871b22-0655-4cc4-9afb-686f9429693f","Type":"ContainerDied","Data":"443f9fd8c62eacb59a359229f88ec2ba45ca0085d46560fc168dd66056393a4d"} Oct 30 12:51:58.627438 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.627407 4413 scope.go:122] "RemoveContainer" containerID="0194586883c3a81b4ec3633f51fca293ab37e9e6ded84c7441425550cbe1d0fe" Oct 30 12:51:58.646038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.646019 4413 scope.go:122] "RemoveContainer" containerID="0194586883c3a81b4ec3633f51fca293ab37e9e6ded84c7441425550cbe1d0fe" Oct 30 12:51:58.646380 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:51:58.646361 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0194586883c3a81b4ec3633f51fca293ab37e9e6ded84c7441425550cbe1d0fe\": container with ID starting with 0194586883c3a81b4ec3633f51fca293ab37e9e6ded84c7441425550cbe1d0fe not found: ID does not exist" containerID="0194586883c3a81b4ec3633f51fca293ab37e9e6ded84c7441425550cbe1d0fe" Oct 30 12:51:58.646439 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.646389 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0194586883c3a81b4ec3633f51fca293ab37e9e6ded84c7441425550cbe1d0fe"} err="failed to get container status \"0194586883c3a81b4ec3633f51fca293ab37e9e6ded84c7441425550cbe1d0fe\": rpc error: code = NotFound desc = could not find container \"0194586883c3a81b4ec3633f51fca293ab37e9e6ded84c7441425550cbe1d0fe\": container with ID starting with 0194586883c3a81b4ec3633f51fca293ab37e9e6ded84c7441425550cbe1d0fe not found: ID does not exist" Oct 30 12:51:58.682342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.682313 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx"] Oct 30 12:51:58.687331 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.687308 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a7148-openshift-default-94c7bfd7c-l5gdx"] Oct 30 12:51:58.707223 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.707187 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k97td\" (UniqueName: \"kubernetes.io/projected/6d871b22-0655-4cc4-9afb-686f9429693f-kube-api-access-k97td\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:51:58.707223 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.707223 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d871b22-0655-4cc4-9afb-686f9429693f-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:51:58.707437 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.707240 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d871b22-0655-4cc4-9afb-686f9429693f-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:51:58.707437 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.707271 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:51:58.707437 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.707284 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d871b22-0655-4cc4-9afb-686f9429693f-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:51:58.971088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:58.971041 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:51:59.030408 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:51:59.030371 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d871b22-0655-4cc4-9afb-686f9429693f" path="/var/lib/kubelet/pods/6d871b22-0655-4cc4-9afb-686f9429693f/volumes" Oct 30 12:52:00.544574 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:00.544543 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:01.136749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:01.136713 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:02.289374 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:02.289337 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:04.052497 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:04.052454 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:05.693634 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:05.693593 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:05.817081 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:52:05.817045 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:52:07.366024 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:07.365988 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:09.292912 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:09.292879 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:11.031633 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:11.031584 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:12.737027 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:12.736937 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:14.576506 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:14.576460 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:16.290028 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:16.289989 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:18.038397 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:18.038357 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:19.675657 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:19.675612 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:21.496652 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:21.496612 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:23.175928 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:23.175839 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:24.883471 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:24.883437 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:26.488465 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:26.488421 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:28.166734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:28.166695 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:29.858719 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:29.858693 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:31.659738 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:31.659710 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:33.347593 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:33.347540 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:35.090883 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:35.090844 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:35.814992 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:52:35.814951 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:52:35.941814 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:35.941787 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:36.055978 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:36.055941 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:36.995541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:36.995511 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:38.226151 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.226118 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx"] Oct 30 12:52:38.226937 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.226922 4413 cpu_manager.go:414] "containerMap: removing container" podUID="6d871b22-0655-4cc4-9afb-686f9429693f" containerName="istio-proxy" Oct 30 12:52:38.226983 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.226938 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d871b22-0655-4cc4-9afb-686f9429693f" containerName="istio-proxy" Oct 30 12:52:38.227075 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.227066 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d871b22-0655-4cc4-9afb-686f9429693f" containerName="istio-proxy" Oct 30 12:52:38.230854 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.230837 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" Oct 30 12:52:38.260123 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.260093 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2545c7\"/\"agc-cred-f4c9839ac4b0\"" Oct 30 12:52:38.266441 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.266408 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx"] Oct 30 12:52:38.407877 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.407844 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zk2t\" (UniqueName: \"kubernetes.io/projected/13f9821e-d4d1-4942-b2f4-0580f8ca9728-kube-api-access-7zk2t\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx\" (UID: \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" Oct 30 12:52:38.408070 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.407913 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f9821e-d4d1-4942-b2f4-0580f8ca9728-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx\" (UID: \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" Oct 30 12:52:38.408070 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.408015 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13f9821e-d4d1-4942-b2f4-0580f8ca9728-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx\" (UID: \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" Oct 30 12:52:38.408163 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.408140 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13f9821e-d4d1-4942-b2f4-0580f8ca9728-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx\" (UID: \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" Oct 30 12:52:38.509711 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.509618 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zk2t\" (UniqueName: \"kubernetes.io/projected/13f9821e-d4d1-4942-b2f4-0580f8ca9728-kube-api-access-7zk2t\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx\" (UID: \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" Oct 30 12:52:38.509880 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.509718 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f9821e-d4d1-4942-b2f4-0580f8ca9728-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx\" (UID: \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" Oct 30 12:52:38.509880 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.509759 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13f9821e-d4d1-4942-b2f4-0580f8ca9728-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx\" (UID: \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" Oct 30 12:52:38.509880 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.509855 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13f9821e-d4d1-4942-b2f4-0580f8ca9728-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx\" (UID: \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" Oct 30 12:52:38.510645 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.510619 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13f9821e-d4d1-4942-b2f4-0580f8ca9728-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx\" (UID: \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" Oct 30 12:52:38.510753 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.510699 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f9821e-d4d1-4942-b2f4-0580f8ca9728-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx\" (UID: \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" Oct 30 12:52:38.512728 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.512708 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13f9821e-d4d1-4942-b2f4-0580f8ca9728-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx\" (UID: \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" Oct 30 12:52:38.533237 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.533208 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zk2t\" (UniqueName: \"kubernetes.io/projected/13f9821e-d4d1-4942-b2f4-0580f8ca9728-kube-api-access-7zk2t\") pod \"payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx\" (UID: \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\") " pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" Oct 30 12:52:38.546181 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.546161 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" Oct 30 12:52:38.852564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.852540 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx"] Oct 30 12:52:38.853305 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:52:38.853278 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f9821e_d4d1_4942_b2f4_0580f8ca9728.slice/crio-45f77240840b125aa771f3ecad455447baef14b9b6f39054b53526d6abd0954e WatchSource:0}: Error finding container 45f77240840b125aa771f3ecad455447baef14b9b6f39054b53526d6abd0954e: Status 404 returned error can't find the container with id 45f77240840b125aa771f3ecad455447baef14b9b6f39054b53526d6abd0954e Oct 30 12:52:38.969948 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.969911 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" event={"ID":"13f9821e-d4d1-4942-b2f4-0580f8ca9728","Type":"ContainerStarted","Data":"5288f55798d2f1b4784d2ac648d414c0e84a728be3796a507f2c161581c2c8bb"} Oct 30 12:52:38.970140 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.970008 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" event={"ID":"13f9821e-d4d1-4942-b2f4-0580f8ca9728","Type":"ContainerStarted","Data":"45f77240840b125aa771f3ecad455447baef14b9b6f39054b53526d6abd0954e"} Oct 30 12:52:38.970211 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.970133 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" Oct 30 12:52:38.996740 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:38.996709 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:39.011523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.011479 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" podStartSLOduration=1.011464161 podStartE2EDuration="1.011464161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:39.009569747 +0000 UTC m=+3512.119412629" watchObservedRunningTime="2026-10-30 12:52:39.011464161 +0000 UTC m=+3512.121307042" Oct 30 12:52:39.079107 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.079076 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5"] Oct 30 12:52:39.084314 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.084290 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" Oct 30 12:52:39.098627 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.098605 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12fd33\"/\"agc-cred-5b0d1b4a6665\"" Oct 30 12:52:39.103133 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.103071 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5"] Oct 30 12:52:39.221858 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.221822 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tclrh\" (UniqueName: \"kubernetes.io/projected/7f2a1284-27cc-4116-b797-f9001308d2eb-kube-api-access-tclrh\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5\" (UID: \"7f2a1284-27cc-4116-b797-f9001308d2eb\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" Oct 30 12:52:39.222060 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.221998 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f2a1284-27cc-4116-b797-f9001308d2eb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5\" (UID: \"7f2a1284-27cc-4116-b797-f9001308d2eb\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" Oct 30 12:52:39.222210 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.222185 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f2a1284-27cc-4116-b797-f9001308d2eb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5\" (UID: \"7f2a1284-27cc-4116-b797-f9001308d2eb\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" Oct 30 12:52:39.222324 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.222224 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f2a1284-27cc-4116-b797-f9001308d2eb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5\" (UID: \"7f2a1284-27cc-4116-b797-f9001308d2eb\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" Oct 30 12:52:39.324098 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.324065 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f2a1284-27cc-4116-b797-f9001308d2eb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5\" (UID: \"7f2a1284-27cc-4116-b797-f9001308d2eb\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" Oct 30 12:52:39.324533 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.324180 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f2a1284-27cc-4116-b797-f9001308d2eb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5\" (UID: \"7f2a1284-27cc-4116-b797-f9001308d2eb\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" Oct 30 12:52:39.324533 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.324201 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f2a1284-27cc-4116-b797-f9001308d2eb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5\" (UID: \"7f2a1284-27cc-4116-b797-f9001308d2eb\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" Oct 30 12:52:39.324533 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.324296 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tclrh\" (UniqueName: \"kubernetes.io/projected/7f2a1284-27cc-4116-b797-f9001308d2eb-kube-api-access-tclrh\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5\" (UID: \"7f2a1284-27cc-4116-b797-f9001308d2eb\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" Oct 30 12:52:39.324899 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.324879 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f2a1284-27cc-4116-b797-f9001308d2eb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5\" (UID: \"7f2a1284-27cc-4116-b797-f9001308d2eb\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" Oct 30 12:52:39.324976 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.324960 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f2a1284-27cc-4116-b797-f9001308d2eb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5\" (UID: \"7f2a1284-27cc-4116-b797-f9001308d2eb\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" Oct 30 12:52:39.326788 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.326768 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f2a1284-27cc-4116-b797-f9001308d2eb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5\" (UID: \"7f2a1284-27cc-4116-b797-f9001308d2eb\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" Oct 30 12:52:39.346805 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.346782 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tclrh\" (UniqueName: \"kubernetes.io/projected/7f2a1284-27cc-4116-b797-f9001308d2eb-kube-api-access-tclrh\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5\" (UID: \"7f2a1284-27cc-4116-b797-f9001308d2eb\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" Oct 30 12:52:39.400086 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.400063 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" Oct 30 12:52:39.709562 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.709532 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5"] Oct 30 12:52:39.710371 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:52:39.710339 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2a1284_27cc_4116_b797_f9001308d2eb.slice/crio-cdd15a50176a9dc8a37092004dfca32419014af8e829b4e592021400151de71c WatchSource:0}: Error finding container cdd15a50176a9dc8a37092004dfca32419014af8e829b4e592021400151de71c: Status 404 returned error can't find the container with id cdd15a50176a9dc8a37092004dfca32419014af8e829b4e592021400151de71c Oct 30 12:52:39.979555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.979461 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" event={"ID":"7f2a1284-27cc-4116-b797-f9001308d2eb","Type":"ContainerStarted","Data":"295785e16ac17f578fbb90ad5f8821660280596b50afd3a8a04509cbea1f23ee"} Oct 30 12:52:39.979555 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.979514 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" event={"ID":"7f2a1284-27cc-4116-b797-f9001308d2eb","Type":"ContainerStarted","Data":"cdd15a50176a9dc8a37092004dfca32419014af8e829b4e592021400151de71c"} Oct 30 12:52:39.979797 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:39.979558 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" Oct 30 12:52:40.023855 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:40.023806 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" podStartSLOduration=1.023789787 podStartE2EDuration="1.023789787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:40.019280531 +0000 UTC m=+3513.129123436" watchObservedRunningTime="2026-10-30 12:52:40.023789787 +0000 UTC m=+3513.133632709" Oct 30 12:52:40.656016 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:40.655978 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:42.267881 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:42.267841 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:43.957454 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:43.957412 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:45.570742 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:45.570704 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:47.272102 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:47.272069 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:48.936900 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:48.936866 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:49.983284 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:49.983234 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" Oct 30 12:52:50.064175 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:50.064142 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw"] Oct 30 12:52:50.465202 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:50.465171 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:50.991113 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:50.991084 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" Oct 30 12:52:51.084815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:51.084784 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk"] Oct 30 12:52:52.249825 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:52.248001 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:54.168154 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:54.168118 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:55.065096 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:55.065050 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" podUID="41e088cd-2edf-4b23-bea8-26dc6a2318ab" containerName="payload-processing" containerID="cri-o://a6018b01c00f969e622d2ae45b9653d68368915fea281494980274454a4452e7" gracePeriod=25 Oct 30 12:52:55.339139 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:55.339082 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" Oct 30 12:52:55.470007 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:55.469977 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41e088cd-2edf-4b23-bea8-26dc6a2318ab-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41e088cd-2edf-4b23-bea8-26dc6a2318ab-routing-overlay-volume\") pod \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\" (UID: \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\") " Oct 30 12:52:55.470208 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:55.470068 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e088cd-2edf-4b23-bea8-26dc6a2318ab-kube-api-access-rbzhp\" (UniqueName: \"kubernetes.io/projected/41e088cd-2edf-4b23-bea8-26dc6a2318ab-kube-api-access-rbzhp\") pod \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\" (UID: \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\") " Oct 30 12:52:55.470208 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:55.470139 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41e088cd-2edf-4b23-bea8-26dc6a2318ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e088cd-2edf-4b23-bea8-26dc6a2318ab-plugins-config-volume\") pod \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\" (UID: \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\") " Oct 30 12:52:55.470366 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:55.470311 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e088cd-2edf-4b23-bea8-26dc6a2318ab-provider-credentials\" (UniqueName: \"kubernetes.io/projected/41e088cd-2edf-4b23-bea8-26dc6a2318ab-provider-credentials\") pod \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\" (UID: \"41e088cd-2edf-4b23-bea8-26dc6a2318ab\") " Oct 30 12:52:55.470510 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:55.470485 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e088cd-2edf-4b23-bea8-26dc6a2318ab-routing-overlay-volume" pod "41e088cd-2edf-4b23-bea8-26dc6a2318ab" (UID: "41e088cd-2edf-4b23-bea8-26dc6a2318ab"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:55.470631 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:55.470611 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e088cd-2edf-4b23-bea8-26dc6a2318ab-plugins-config-volume" pod "41e088cd-2edf-4b23-bea8-26dc6a2318ab" (UID: "41e088cd-2edf-4b23-bea8-26dc6a2318ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:55.470986 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:55.470910 4413 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41e088cd-2edf-4b23-bea8-26dc6a2318ab-routing-overlay-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:52:55.470986 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:55.470932 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e088cd-2edf-4b23-bea8-26dc6a2318ab-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:52:55.472523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:55.472503 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e088cd-2edf-4b23-bea8-26dc6a2318ab-kube-api-access-rbzhp" pod "41e088cd-2edf-4b23-bea8-26dc6a2318ab" (UID: "41e088cd-2edf-4b23-bea8-26dc6a2318ab"). InnerVolumeSpecName "kube-api-access-rbzhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:55.472622 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:55.472530 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e088cd-2edf-4b23-bea8-26dc6a2318ab-provider-credentials" pod "41e088cd-2edf-4b23-bea8-26dc6a2318ab" (UID: "41e088cd-2edf-4b23-bea8-26dc6a2318ab"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:55.572941 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:55.572904 4413 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41e088cd-2edf-4b23-bea8-26dc6a2318ab-provider-credentials\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:52:55.572941 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:55.572933 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbzhp\" (UniqueName: \"kubernetes.io/projected/41e088cd-2edf-4b23-bea8-26dc6a2318ab-kube-api-access-rbzhp\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:52:55.874964 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:55.874913 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:56.085961 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.085905 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" podUID="22fd6023-a535-4f01-ace9-593c8381536e" containerName="payload-processing" containerID="cri-o://47ed23327b4473bbd19ad76d7806ff334fa317e39c6769d8207f5db22faf934b" gracePeriod=25 Oct 30 12:52:56.111138 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.111106 4413 generic.go:356] "Generic (PLEG): container finished" podID="41e088cd-2edf-4b23-bea8-26dc6a2318ab" containerID="a6018b01c00f969e622d2ae45b9653d68368915fea281494980274454a4452e7" exitCode=0 Oct 30 12:52:56.111290 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.111145 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" Oct 30 12:52:56.111290 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.111176 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" event={"ID":"41e088cd-2edf-4b23-bea8-26dc6a2318ab","Type":"ContainerDied","Data":"a6018b01c00f969e622d2ae45b9653d68368915fea281494980274454a4452e7"} Oct 30 12:52:56.111290 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.111236 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw" event={"ID":"41e088cd-2edf-4b23-bea8-26dc6a2318ab","Type":"ContainerDied","Data":"6b14268dd9a608dde00578c56f541a6f22c6978b95b0235f56dd861e4759aa22"} Oct 30 12:52:56.111419 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.111305 4413 scope.go:122] "RemoveContainer" containerID="a6018b01c00f969e622d2ae45b9653d68368915fea281494980274454a4452e7" Oct 30 12:52:56.201977 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.201949 4413 scope.go:122] "RemoveContainer" containerID="a6018b01c00f969e622d2ae45b9653d68368915fea281494980274454a4452e7" Oct 30 12:52:56.202345 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:52:56.202324 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6018b01c00f969e622d2ae45b9653d68368915fea281494980274454a4452e7\": container with ID starting with a6018b01c00f969e622d2ae45b9653d68368915fea281494980274454a4452e7 not found: ID does not exist" containerID="a6018b01c00f969e622d2ae45b9653d68368915fea281494980274454a4452e7" Oct 30 12:52:56.202446 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.202353 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6018b01c00f969e622d2ae45b9653d68368915fea281494980274454a4452e7"} err="failed to get container status \"a6018b01c00f969e622d2ae45b9653d68368915fea281494980274454a4452e7\": rpc error: code = NotFound desc = could not find container \"a6018b01c00f969e622d2ae45b9653d68368915fea281494980274454a4452e7\": container with ID starting with a6018b01c00f969e622d2ae45b9653d68368915fea281494980274454a4452e7 not found: ID does not exist" Oct 30 12:52:56.202678 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.202657 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw"] Oct 30 12:52:56.213537 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.213512 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-6d84v4bfw"] Oct 30 12:52:56.366890 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.366867 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" Oct 30 12:52:56.383621 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.383586 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22fd6023-a535-4f01-ace9-593c8381536e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fd6023-a535-4f01-ace9-593c8381536e-plugins-config-volume\") pod \"22fd6023-a535-4f01-ace9-593c8381536e\" (UID: \"22fd6023-a535-4f01-ace9-593c8381536e\") " Oct 30 12:52:56.383761 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.383653 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fd6023-a535-4f01-ace9-593c8381536e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/22fd6023-a535-4f01-ace9-593c8381536e-provider-credentials\") pod \"22fd6023-a535-4f01-ace9-593c8381536e\" (UID: \"22fd6023-a535-4f01-ace9-593c8381536e\") " Oct 30 12:52:56.383829 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.383796 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22fd6023-a535-4f01-ace9-593c8381536e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22fd6023-a535-4f01-ace9-593c8381536e-routing-overlay-volume\") pod \"22fd6023-a535-4f01-ace9-593c8381536e\" (UID: \"22fd6023-a535-4f01-ace9-593c8381536e\") " Oct 30 12:52:56.383923 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.383905 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fd6023-a535-4f01-ace9-593c8381536e-kube-api-access-wdcdk\" (UniqueName: \"kubernetes.io/projected/22fd6023-a535-4f01-ace9-593c8381536e-kube-api-access-wdcdk\") pod \"22fd6023-a535-4f01-ace9-593c8381536e\" (UID: \"22fd6023-a535-4f01-ace9-593c8381536e\") " Oct 30 12:52:56.384129 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.384106 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fd6023-a535-4f01-ace9-593c8381536e-plugins-config-volume" pod "22fd6023-a535-4f01-ace9-593c8381536e" (UID: "22fd6023-a535-4f01-ace9-593c8381536e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:56.384129 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.384114 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fd6023-a535-4f01-ace9-593c8381536e-routing-overlay-volume" pod "22fd6023-a535-4f01-ace9-593c8381536e" (UID: "22fd6023-a535-4f01-ace9-593c8381536e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:52:56.384550 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.384530 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fd6023-a535-4f01-ace9-593c8381536e-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:52:56.384609 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.384557 4413 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22fd6023-a535-4f01-ace9-593c8381536e-routing-overlay-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:52:56.386204 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.386181 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fd6023-a535-4f01-ace9-593c8381536e-kube-api-access-wdcdk" pod "22fd6023-a535-4f01-ace9-593c8381536e" (UID: "22fd6023-a535-4f01-ace9-593c8381536e"). InnerVolumeSpecName "kube-api-access-wdcdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:56.386204 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.386197 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fd6023-a535-4f01-ace9-593c8381536e-provider-credentials" pod "22fd6023-a535-4f01-ace9-593c8381536e" (UID: "22fd6023-a535-4f01-ace9-593c8381536e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:52:56.486071 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.486001 4413 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22fd6023-a535-4f01-ace9-593c8381536e-provider-credentials\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:52:56.486071 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.486030 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdcdk\" (UniqueName: \"kubernetes.io/projected/22fd6023-a535-4f01-ace9-593c8381536e-kube-api-access-wdcdk\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:52:56.940690 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:56.940663 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:57.023886 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:57.023854 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e088cd-2edf-4b23-bea8-26dc6a2318ab" path="/var/lib/kubelet/pods/41e088cd-2edf-4b23-bea8-26dc6a2318ab/volumes" Oct 30 12:52:57.050186 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:57.050161 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:57.123113 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:57.123082 4413 generic.go:356] "Generic (PLEG): container finished" podID="22fd6023-a535-4f01-ace9-593c8381536e" containerID="47ed23327b4473bbd19ad76d7806ff334fa317e39c6769d8207f5db22faf934b" exitCode=0 Oct 30 12:52:57.123113 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:57.123118 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" event={"ID":"22fd6023-a535-4f01-ace9-593c8381536e","Type":"ContainerDied","Data":"47ed23327b4473bbd19ad76d7806ff334fa317e39c6769d8207f5db22faf934b"} Oct 30 12:52:57.123423 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:57.123140 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" Oct 30 12:52:57.123423 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:57.123165 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk" event={"ID":"22fd6023-a535-4f01-ace9-593c8381536e","Type":"ContainerDied","Data":"bdb16a853292ef941d0e8f51f6386d7097e8c14302f1351c1557162dbfb2dad2"} Oct 30 12:52:57.123423 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:57.123189 4413 scope.go:122] "RemoveContainer" containerID="47ed23327b4473bbd19ad76d7806ff334fa317e39c6769d8207f5db22faf934b" Oct 30 12:52:57.134679 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:57.134655 4413 scope.go:122] "RemoveContainer" containerID="47ed23327b4473bbd19ad76d7806ff334fa317e39c6769d8207f5db22faf934b" Oct 30 12:52:57.135786 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:52:57.135746 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ed23327b4473bbd19ad76d7806ff334fa317e39c6769d8207f5db22faf934b\": container with ID starting with 47ed23327b4473bbd19ad76d7806ff334fa317e39c6769d8207f5db22faf934b not found: ID does not exist" containerID="47ed23327b4473bbd19ad76d7806ff334fa317e39c6769d8207f5db22faf934b" Oct 30 12:52:57.135897 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:57.135781 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ed23327b4473bbd19ad76d7806ff334fa317e39c6769d8207f5db22faf934b"} err="failed to get container status \"47ed23327b4473bbd19ad76d7806ff334fa317e39c6769d8207f5db22faf934b\": rpc error: code = NotFound desc = could not find container \"47ed23327b4473bbd19ad76d7806ff334fa317e39c6769d8207f5db22faf934b\": container with ID starting with 47ed23327b4473bbd19ad76d7806ff334fa317e39c6769d8207f5db22faf934b not found: ID does not exist" Oct 30 12:52:57.177749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:57.177706 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk"] Oct 30 12:52:57.189502 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:57.189468 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76fnqzkk"] Oct 30 12:52:57.439312 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:57.439275 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:52:59.021989 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:59.021957 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22fd6023-a535-4f01-ace9-593c8381536e" path="/var/lib/kubelet/pods/22fd6023-a535-4f01-ace9-593c8381536e/volumes" Oct 30 12:52:59.188565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:52:59.188533 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:00.867662 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:00.867613 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:01.249033 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:01.249005 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:02.428877 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:02.428844 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:04.168969 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:04.168938 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:05.815449 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:53:05.815417 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:53:05.880267 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:05.880222 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:07.431196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:07.431150 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:09.025760 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:09.025716 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:10.676906 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:10.676877 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:12.255038 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:12.255004 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:13.901833 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:13.901801 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:15.666277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:15.666222 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:17.363355 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:17.363324 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:19.025003 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:19.024922 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:20.675061 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:20.675013 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:22.347846 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:22.347813 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:24.102775 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:24.102742 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:25.791920 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:25.791895 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:27.471868 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:27.471840 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:29.163008 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:29.162967 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:30.762579 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:30.762544 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:32.414826 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:32.414773 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:34.159447 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:34.159405 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:35.815952 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:53:35.815914 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:53:35.860571 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:35.860542 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:37.572267 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:37.572224 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:39.264266 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:39.264216 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:40.863856 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:40.863825 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:42.420175 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:42.420146 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:44.203768 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:44.203737 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:45.106351 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.106318 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:45.254359 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.254320 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:45.428036 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.428003 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl"] Oct 30 12:53:45.429121 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.429098 4413 cpu_manager.go:414] "containerMap: removing container" podUID="22fd6023-a535-4f01-ace9-593c8381536e" containerName="payload-processing" Oct 30 12:53:45.429194 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.429125 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fd6023-a535-4f01-ace9-593c8381536e" containerName="payload-processing" Oct 30 12:53:45.429194 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.429191 4413 cpu_manager.go:414] "containerMap: removing container" podUID="41e088cd-2edf-4b23-bea8-26dc6a2318ab" containerName="payload-processing" Oct 30 12:53:45.429303 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.429200 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e088cd-2edf-4b23-bea8-26dc6a2318ab" containerName="payload-processing" Oct 30 12:53:45.429415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.429402 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fd6023-a535-4f01-ace9-593c8381536e" containerName="payload-processing" Oct 30 12:53:45.429455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.429428 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e088cd-2edf-4b23-bea8-26dc6a2318ab" containerName="payload-processing" Oct 30 12:53:45.433821 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.433801 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" Oct 30 12:53:45.454407 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.454377 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl"] Oct 30 12:53:45.461795 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.461767 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-57f1b32b361d\"" Oct 30 12:53:45.461956 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.461869 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 12:53:45.461956 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.461905 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 12:53:45.462078 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.461983 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4ff758d7dea0\"" Oct 30 12:53:45.462078 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.461995 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-4ktzx\"" Oct 30 12:53:45.462195 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.462091 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 12:53:45.462414 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.462395 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 12:53:45.587179 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.587151 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scdt4\" (UniqueName: \"kubernetes.io/projected/56af769c-6f94-44de-ad72-b0e406e531a8-kube-api-access-scdt4\") pod \"payload-processing-external-model-6c9884c959-7qlvl\" (UID: \"56af769c-6f94-44de-ad72-b0e406e531a8\") " pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" Oct 30 12:53:45.587388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.587309 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56af769c-6f94-44de-ad72-b0e406e531a8-provider-credentials\") pod \"payload-processing-external-model-6c9884c959-7qlvl\" (UID: \"56af769c-6f94-44de-ad72-b0e406e531a8\") " pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" Oct 30 12:53:45.587388 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.587362 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56af769c-6f94-44de-ad72-b0e406e531a8-plugins-config-volume\") pod \"payload-processing-external-model-6c9884c959-7qlvl\" (UID: \"56af769c-6f94-44de-ad72-b0e406e531a8\") " pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" Oct 30 12:53:45.587627 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.587608 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56af769c-6f94-44de-ad72-b0e406e531a8-routing-overlay-volume\") pod \"payload-processing-external-model-6c9884c959-7qlvl\" (UID: \"56af769c-6f94-44de-ad72-b0e406e531a8\") " pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" Oct 30 12:53:45.689502 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.689428 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scdt4\" (UniqueName: \"kubernetes.io/projected/56af769c-6f94-44de-ad72-b0e406e531a8-kube-api-access-scdt4\") pod \"payload-processing-external-model-6c9884c959-7qlvl\" (UID: \"56af769c-6f94-44de-ad72-b0e406e531a8\") " pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" Oct 30 12:53:45.689686 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.689517 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56af769c-6f94-44de-ad72-b0e406e531a8-provider-credentials\") pod \"payload-processing-external-model-6c9884c959-7qlvl\" (UID: \"56af769c-6f94-44de-ad72-b0e406e531a8\") " pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" Oct 30 12:53:45.689686 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.689561 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56af769c-6f94-44de-ad72-b0e406e531a8-plugins-config-volume\") pod \"payload-processing-external-model-6c9884c959-7qlvl\" (UID: \"56af769c-6f94-44de-ad72-b0e406e531a8\") " pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" Oct 30 12:53:45.689820 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.689696 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56af769c-6f94-44de-ad72-b0e406e531a8-routing-overlay-volume\") pod \"payload-processing-external-model-6c9884c959-7qlvl\" (UID: \"56af769c-6f94-44de-ad72-b0e406e531a8\") " pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" Oct 30 12:53:45.690323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.690304 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56af769c-6f94-44de-ad72-b0e406e531a8-routing-overlay-volume\") pod \"payload-processing-external-model-6c9884c959-7qlvl\" (UID: \"56af769c-6f94-44de-ad72-b0e406e531a8\") " pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" Oct 30 12:53:45.690448 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.690431 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56af769c-6f94-44de-ad72-b0e406e531a8-plugins-config-volume\") pod \"payload-processing-external-model-6c9884c959-7qlvl\" (UID: \"56af769c-6f94-44de-ad72-b0e406e531a8\") " pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" Oct 30 12:53:45.692182 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.692159 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56af769c-6f94-44de-ad72-b0e406e531a8-provider-credentials\") pod \"payload-processing-external-model-6c9884c959-7qlvl\" (UID: \"56af769c-6f94-44de-ad72-b0e406e531a8\") " pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" Oct 30 12:53:45.712749 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.712714 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scdt4\" (UniqueName: \"kubernetes.io/projected/56af769c-6f94-44de-ad72-b0e406e531a8-kube-api-access-scdt4\") pod \"payload-processing-external-model-6c9884c959-7qlvl\" (UID: \"56af769c-6f94-44de-ad72-b0e406e531a8\") " pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" Oct 30 12:53:45.749358 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:45.749323 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" Oct 30 12:53:46.060932 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:46.060899 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl"] Oct 30 12:53:46.061977 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:53:46.061948 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56af769c_6f94_44de_ad72_b0e406e531a8.slice/crio-51df789ff32aa4ef8e842237cbe74a27231dfcfa11a00a51fbffbe0078ea3361 WatchSource:0}: Error finding container 51df789ff32aa4ef8e842237cbe74a27231dfcfa11a00a51fbffbe0078ea3361: Status 404 returned error can't find the container with id 51df789ff32aa4ef8e842237cbe74a27231dfcfa11a00a51fbffbe0078ea3361 Oct 30 12:53:46.244090 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:46.244057 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:46.506634 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:46.506601 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" event={"ID":"56af769c-6f94-44de-ad72-b0e406e531a8","Type":"ContainerStarted","Data":"6d9163a713ac0a508a0ed3a3d0b211a265ed0eae964d6b21fa0ab90852eafba3"} Oct 30 12:53:46.507055 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:46.506656 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" event={"ID":"56af769c-6f94-44de-ad72-b0e406e531a8","Type":"ContainerStarted","Data":"51df789ff32aa4ef8e842237cbe74a27231dfcfa11a00a51fbffbe0078ea3361"} Oct 30 12:53:46.507055 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:46.506730 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" Oct 30 12:53:46.546802 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:46.546756 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" podStartSLOduration=1.546744967 podStartE2EDuration="1.546744967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:46.546568986 +0000 UTC m=+3579.656411866" watchObservedRunningTime="2026-10-30 12:53:46.546744967 +0000 UTC m=+3579.656587848" Oct 30 12:53:47.241307 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:47.241274 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:47.666902 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:47.666863 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:47.969833 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:47.969718 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:48.322718 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:48.322583 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:48.505808 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:48.505779 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:50.055035 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:50.054999 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:51.368591 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:51.368551 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:51.492411 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:51.492382 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:52.138240 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:52.138205 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:53.024768 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.024736 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:53.139003 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.138968 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:53.188724 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.188692 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x"] Oct 30 12:53:53.196583 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.196560 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" Oct 30 12:53:53.212427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.212398 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x"] Oct 30 12:53:53.214524 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.214499 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb8871c-9f64-44a2-bc8e-c110b777d677-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x\" (UID: \"4fb8871c-9f64-44a2-bc8e-c110b777d677\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" Oct 30 12:53:53.214752 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.214732 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk7cr\" (UniqueName: \"kubernetes.io/projected/4fb8871c-9f64-44a2-bc8e-c110b777d677-kube-api-access-pk7cr\") pod \"payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x\" (UID: \"4fb8871c-9f64-44a2-bc8e-c110b777d677\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" Oct 30 12:53:53.219221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.219195 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-eab799\"" Oct 30 12:53:53.219221 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.219205 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-eab799-dockercfg-w7v5j\"" Oct 30 12:53:53.332323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.328361 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb8871c-9f64-44a2-bc8e-c110b777d677-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x\" (UID: \"4fb8871c-9f64-44a2-bc8e-c110b777d677\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" Oct 30 12:53:53.332323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.328607 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk7cr\" (UniqueName: \"kubernetes.io/projected/4fb8871c-9f64-44a2-bc8e-c110b777d677-kube-api-access-pk7cr\") pod \"payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x\" (UID: \"4fb8871c-9f64-44a2-bc8e-c110b777d677\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" Oct 30 12:53:53.332323 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.329892 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb8871c-9f64-44a2-bc8e-c110b777d677-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x\" (UID: \"4fb8871c-9f64-44a2-bc8e-c110b777d677\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" Oct 30 12:53:53.364595 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.364563 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf"] Oct 30 12:53:53.384605 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.384556 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk7cr\" (UniqueName: \"kubernetes.io/projected/4fb8871c-9f64-44a2-bc8e-c110b777d677-kube-api-access-pk7cr\") pod \"payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x\" (UID: \"4fb8871c-9f64-44a2-bc8e-c110b777d677\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" Oct 30 12:53:53.414915 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.414876 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf"] Oct 30 12:53:53.415117 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.415049 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf" Oct 30 12:53:53.516884 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.516852 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" Oct 30 12:53:53.532195 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.532156 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaacfb-1b6b-4cde-854b-95fd32c6dce2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf\" (UID: \"50eaacfb-1b6b-4cde-854b-95fd32c6dce2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf" Oct 30 12:53:53.532502 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.532480 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzww2\" (UniqueName: \"kubernetes.io/projected/50eaacfb-1b6b-4cde-854b-95fd32c6dce2-kube-api-access-wzww2\") pod \"payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf\" (UID: \"50eaacfb-1b6b-4cde-854b-95fd32c6dce2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf" Oct 30 12:53:53.634654 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.634600 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzww2\" (UniqueName: \"kubernetes.io/projected/50eaacfb-1b6b-4cde-854b-95fd32c6dce2-kube-api-access-wzww2\") pod \"payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf\" (UID: \"50eaacfb-1b6b-4cde-854b-95fd32c6dce2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf" Oct 30 12:53:53.634842 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.634734 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaacfb-1b6b-4cde-854b-95fd32c6dce2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf\" (UID: \"50eaacfb-1b6b-4cde-854b-95fd32c6dce2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf" Oct 30 12:53:53.635695 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.635669 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaacfb-1b6b-4cde-854b-95fd32c6dce2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf\" (UID: \"50eaacfb-1b6b-4cde-854b-95fd32c6dce2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf" Oct 30 12:53:53.659362 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.659331 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzww2\" (UniqueName: \"kubernetes.io/projected/50eaacfb-1b6b-4cde-854b-95fd32c6dce2-kube-api-access-wzww2\") pod \"payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf\" (UID: \"50eaacfb-1b6b-4cde-854b-95fd32c6dce2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf" Oct 30 12:53:53.733089 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.733044 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf" Oct 30 12:53:53.829242 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.829212 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x"] Oct 30 12:53:53.830100 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:53:53.830066 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb8871c_9f64_44a2_bc8e_c110b777d677.slice/crio-65bd78b367a66141049dd8fffc5a084e4f1c36ac34e6a0f889bd1bd8b46769c6 WatchSource:0}: Error finding container 65bd78b367a66141049dd8fffc5a084e4f1c36ac34e6a0f889bd1bd8b46769c6: Status 404 returned error can't find the container with id 65bd78b367a66141049dd8fffc5a084e4f1c36ac34e6a0f889bd1bd8b46769c6 Oct 30 12:53:53.860740 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:53.860704 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:54.044041 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:54.044010 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf"] Oct 30 12:53:54.045164 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:53:54.045135 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50eaacfb_1b6b_4cde_854b_95fd32c6dce2.slice/crio-09e4ffea0acacaa7a94476a63fafcb3ebbc4f8ce8fa51280416ea3ae7dfa8b25 WatchSource:0}: Error finding container 09e4ffea0acacaa7a94476a63fafcb3ebbc4f8ce8fa51280416ea3ae7dfa8b25: Status 404 returned error can't find the container with id 09e4ffea0acacaa7a94476a63fafcb3ebbc4f8ce8fa51280416ea3ae7dfa8b25 Oct 30 12:53:54.570403 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:54.570359 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf" event={"ID":"50eaacfb-1b6b-4cde-854b-95fd32c6dce2","Type":"ContainerStarted","Data":"78722c88d588b80270c8792473c0fe94d39e3ff0a86ca4975d2a565741615e0a"} Oct 30 12:53:54.570600 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:54.570428 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf" event={"ID":"50eaacfb-1b6b-4cde-854b-95fd32c6dce2","Type":"ContainerStarted","Data":"09e4ffea0acacaa7a94476a63fafcb3ebbc4f8ce8fa51280416ea3ae7dfa8b25"} Oct 30 12:53:54.570600 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:54.570458 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf" Oct 30 12:53:54.572101 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:54.572079 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" event={"ID":"4fb8871c-9f64-44a2-bc8e-c110b777d677","Type":"ContainerStarted","Data":"d1b3fb515a5f7b4cbf8bb40528ce47462ba8e979e67bb55563e087241879ac8c"} Oct 30 12:53:54.572235 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:54.572121 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" event={"ID":"4fb8871c-9f64-44a2-bc8e-c110b777d677","Type":"ContainerStarted","Data":"65bd78b367a66141049dd8fffc5a084e4f1c36ac34e6a0f889bd1bd8b46769c6"} Oct 30 12:53:54.572235 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:54.572219 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" Oct 30 12:53:54.613813 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:54.613765 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf" podStartSLOduration=1.6137507439999998 podStartE2EDuration="1.613750744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:54.610024377 +0000 UTC m=+3587.719867257" watchObservedRunningTime="2026-10-30 12:53:54.613750744 +0000 UTC m=+3587.723593625" Oct 30 12:53:54.652058 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:54.652010 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" podStartSLOduration=1.6519965189999999 podStartE2EDuration="1.651996519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:54.650900211 +0000 UTC m=+3587.760743093" watchObservedRunningTime="2026-10-30 12:53:54.651996519 +0000 UTC m=+3587.761839401" Oct 30 12:53:55.459030 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:55.458995 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:57.102832 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:57.102803 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:53:57.518002 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:57.517973 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" Oct 30 12:53:58.766130 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:53:58.766096 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:00.444996 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:00.444966 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:01.400635 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:01.400602 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:02.079464 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:02.079419 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:03.766140 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:03.766077 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:05.452842 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:05.452804 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:05.582775 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:05.582748 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf" Oct 30 12:54:05.583168 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:05.583146 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" Oct 30 12:54:05.815623 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:54:05.815535 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:54:07.043140 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:07.043101 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:08.155830 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:08.155801 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:08.282275 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:08.278239 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:09.070357 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:09.070319 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:09.442620 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:09.442586 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz"] Oct 30 12:54:09.448703 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:09.448684 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" Oct 30 12:54:09.467708 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:09.467684 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz"] Oct 30 12:54:09.504556 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:09.504505 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gnwz\" (UniqueName: \"kubernetes.io/projected/76533887-6247-4a26-9bde-650e13659209-kube-api-access-2gnwz\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz\" (UID: \"76533887-6247-4a26-9bde-650e13659209\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" Oct 30 12:54:09.504742 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:09.504717 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76533887-6247-4a26-9bde-650e13659209-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz\" (UID: \"76533887-6247-4a26-9bde-650e13659209\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" Oct 30 12:54:09.504864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:09.504837 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76533887-6247-4a26-9bde-650e13659209-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz\" (UID: \"76533887-6247-4a26-9bde-650e13659209\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" Oct 30 12:54:09.505148 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:09.505130 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76533887-6247-4a26-9bde-650e13659209-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz\" (UID: \"76533887-6247-4a26-9bde-650e13659209\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" Oct 30 12:54:09.606484 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:09.606445 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gnwz\" (UniqueName: \"kubernetes.io/projected/76533887-6247-4a26-9bde-650e13659209-kube-api-access-2gnwz\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz\" (UID: \"76533887-6247-4a26-9bde-650e13659209\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" Oct 30 12:54:09.606700 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:09.606589 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76533887-6247-4a26-9bde-650e13659209-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz\" (UID: \"76533887-6247-4a26-9bde-650e13659209\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" Oct 30 12:54:09.606700 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:09.606642 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76533887-6247-4a26-9bde-650e13659209-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz\" (UID: \"76533887-6247-4a26-9bde-650e13659209\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" Oct 30 12:54:09.606922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:09.606897 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76533887-6247-4a26-9bde-650e13659209-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz\" (UID: \"76533887-6247-4a26-9bde-650e13659209\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" Oct 30 12:54:09.607474 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:09.607449 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76533887-6247-4a26-9bde-650e13659209-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz\" (UID: \"76533887-6247-4a26-9bde-650e13659209\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" Oct 30 12:54:09.607577 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:09.607472 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76533887-6247-4a26-9bde-650e13659209-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz\" (UID: \"76533887-6247-4a26-9bde-650e13659209\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" Oct 30 12:54:09.609409 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:09.609383 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76533887-6247-4a26-9bde-650e13659209-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz\" (UID: \"76533887-6247-4a26-9bde-650e13659209\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" Oct 30 12:54:09.630041 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:09.630003 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gnwz\" (UniqueName: \"kubernetes.io/projected/76533887-6247-4a26-9bde-650e13659209-kube-api-access-2gnwz\") pod \"payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz\" (UID: \"76533887-6247-4a26-9bde-650e13659209\") " pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" Oct 30 12:54:09.764063 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:09.763985 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" Oct 30 12:54:10.070712 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:10.070687 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz"] Oct 30 12:54:10.071169 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:54:10.071141 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76533887_6247_4a26_9bde_650e13659209.slice/crio-23879a166b3c066b7104dc52d6c41e940a4d09a5af60929c1d2806204d835e6b WatchSource:0}: Error finding container 23879a166b3c066b7104dc52d6c41e940a4d09a5af60929c1d2806204d835e6b: Status 404 returned error can't find the container with id 23879a166b3c066b7104dc52d6c41e940a4d09a5af60929c1d2806204d835e6b Oct 30 12:54:10.716243 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:10.716201 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" event={"ID":"76533887-6247-4a26-9bde-650e13659209","Type":"ContainerStarted","Data":"3ded870740ec088dc2f4c751f1b217303dcbb94f938761f801627e46066631c2"} Oct 30 12:54:10.716752 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:10.716336 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" Oct 30 12:54:10.716752 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:10.716355 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" event={"ID":"76533887-6247-4a26-9bde-650e13659209","Type":"ContainerStarted","Data":"23879a166b3c066b7104dc52d6c41e940a4d09a5af60929c1d2806204d835e6b"} Oct 30 12:54:10.759576 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:10.759515 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" podStartSLOduration=1.7594968949999998 podStartE2EDuration="1.759496895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:10.756183179 +0000 UTC m=+3603.866026061" watchObservedRunningTime="2026-10-30 12:54:10.759496895 +0000 UTC m=+3603.869339777" Oct 30 12:54:10.777500 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:10.777460 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:12.400895 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:12.400853 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:14.043583 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:14.043554 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:15.691648 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:15.691617 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:17.565887 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:17.565823 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:19.833420 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:19.833386 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:21.447340 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:21.447308 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx"] Oct 30 12:54:21.624746 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:21.624703 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:21.727530 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:21.727447 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" Oct 30 12:54:21.809416 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:21.809381 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5"] Oct 30 12:54:23.332693 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:23.332603 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:25.121864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:25.121826 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:26.448424 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.448382 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" podUID="13f9821e-d4d1-4942-b2f4-0580f8ca9728" containerName="payload-processing" containerID="cri-o://5288f55798d2f1b4784d2ac648d414c0e84a728be3796a507f2c161581c2c8bb" gracePeriod=25 Oct 30 12:54:26.729145 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.729118 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" Oct 30 12:54:26.771843 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.771807 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:26.794734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.794694 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f9821e-d4d1-4942-b2f4-0580f8ca9728-provider-credentials\" (UniqueName: \"kubernetes.io/projected/13f9821e-d4d1-4942-b2f4-0580f8ca9728-provider-credentials\") pod \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\" (UID: \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\") " Oct 30 12:54:26.794941 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.794783 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13f9821e-d4d1-4942-b2f4-0580f8ca9728-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13f9821e-d4d1-4942-b2f4-0580f8ca9728-routing-overlay-volume\") pod \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\" (UID: \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\") " Oct 30 12:54:26.794941 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.794912 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13f9821e-d4d1-4942-b2f4-0580f8ca9728-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f9821e-d4d1-4942-b2f4-0580f8ca9728-plugins-config-volume\") pod \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\" (UID: \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\") " Oct 30 12:54:26.795092 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.794950 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f9821e-d4d1-4942-b2f4-0580f8ca9728-kube-api-access-7zk2t\" (UniqueName: \"kubernetes.io/projected/13f9821e-d4d1-4942-b2f4-0580f8ca9728-kube-api-access-7zk2t\") pod \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\" (UID: \"13f9821e-d4d1-4942-b2f4-0580f8ca9728\") " Oct 30 12:54:26.796449 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.796421 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f9821e-d4d1-4942-b2f4-0580f8ca9728-routing-overlay-volume" pod "13f9821e-d4d1-4942-b2f4-0580f8ca9728" (UID: "13f9821e-d4d1-4942-b2f4-0580f8ca9728"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:54:26.796709 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.796682 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f9821e-d4d1-4942-b2f4-0580f8ca9728-plugins-config-volume" pod "13f9821e-d4d1-4942-b2f4-0580f8ca9728" (UID: "13f9821e-d4d1-4942-b2f4-0580f8ca9728"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:54:26.799010 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.798988 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f9821e-d4d1-4942-b2f4-0580f8ca9728-kube-api-access-7zk2t" pod "13f9821e-d4d1-4942-b2f4-0580f8ca9728" (UID: "13f9821e-d4d1-4942-b2f4-0580f8ca9728"). InnerVolumeSpecName "kube-api-access-7zk2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:26.799131 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.799076 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f9821e-d4d1-4942-b2f4-0580f8ca9728-provider-credentials" pod "13f9821e-d4d1-4942-b2f4-0580f8ca9728" (UID: "13f9821e-d4d1-4942-b2f4-0580f8ca9728"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:26.810945 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.810912 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" podUID="7f2a1284-27cc-4116-b797-f9001308d2eb" containerName="payload-processing" containerID="cri-o://295785e16ac17f578fbb90ad5f8821660280596b50afd3a8a04509cbea1f23ee" gracePeriod=25 Oct 30 12:54:26.855987 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.855955 4413 generic.go:356] "Generic (PLEG): container finished" podID="13f9821e-d4d1-4942-b2f4-0580f8ca9728" containerID="5288f55798d2f1b4784d2ac648d414c0e84a728be3796a507f2c161581c2c8bb" exitCode=0 Oct 30 12:54:26.856142 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.856005 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" Oct 30 12:54:26.856142 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.856030 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" event={"ID":"13f9821e-d4d1-4942-b2f4-0580f8ca9728","Type":"ContainerDied","Data":"5288f55798d2f1b4784d2ac648d414c0e84a728be3796a507f2c161581c2c8bb"} Oct 30 12:54:26.856142 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.856088 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx" event={"ID":"13f9821e-d4d1-4942-b2f4-0580f8ca9728","Type":"ContainerDied","Data":"45f77240840b125aa771f3ecad455447baef14b9b6f39054b53526d6abd0954e"} Oct 30 12:54:26.856142 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.856111 4413 scope.go:122] "RemoveContainer" containerID="5288f55798d2f1b4784d2ac648d414c0e84a728be3796a507f2c161581c2c8bb" Oct 30 12:54:26.897765 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.897734 4413 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13f9821e-d4d1-4942-b2f4-0580f8ca9728-provider-credentials\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:54:26.897904 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.897768 4413 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13f9821e-d4d1-4942-b2f4-0580f8ca9728-routing-overlay-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:54:26.897904 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.897784 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f9821e-d4d1-4942-b2f4-0580f8ca9728-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:54:26.897904 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.897797 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zk2t\" (UniqueName: \"kubernetes.io/projected/13f9821e-d4d1-4942-b2f4-0580f8ca9728-kube-api-access-7zk2t\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:54:26.938396 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.938346 4413 scope.go:122] "RemoveContainer" containerID="5288f55798d2f1b4784d2ac648d414c0e84a728be3796a507f2c161581c2c8bb" Oct 30 12:54:26.938783 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:54:26.938748 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5288f55798d2f1b4784d2ac648d414c0e84a728be3796a507f2c161581c2c8bb\": container with ID starting with 5288f55798d2f1b4784d2ac648d414c0e84a728be3796a507f2c161581c2c8bb not found: ID does not exist" containerID="5288f55798d2f1b4784d2ac648d414c0e84a728be3796a507f2c161581c2c8bb" Oct 30 12:54:26.938999 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.938792 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5288f55798d2f1b4784d2ac648d414c0e84a728be3796a507f2c161581c2c8bb"} err="failed to get container status \"5288f55798d2f1b4784d2ac648d414c0e84a728be3796a507f2c161581c2c8bb\": rpc error: code = NotFound desc = could not find container \"5288f55798d2f1b4784d2ac648d414c0e84a728be3796a507f2c161581c2c8bb\": container with ID starting with 5288f55798d2f1b4784d2ac648d414c0e84a728be3796a507f2c161581c2c8bb not found: ID does not exist" Oct 30 12:54:26.941519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.941493 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx"] Oct 30 12:54:26.945430 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:26.945409 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2545c7/payload-processing-external-model-e2e-extmod-a-2545c7-897dvswmx"] Oct 30 12:54:27.036801 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.036770 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13f9821e-d4d1-4942-b2f4-0580f8ca9728" path="/var/lib/kubelet/pods/13f9821e-d4d1-4942-b2f4-0580f8ca9728/volumes" Oct 30 12:54:27.102320 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.102293 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" Oct 30 12:54:27.203236 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.203203 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f2a1284-27cc-4116-b797-f9001308d2eb-kube-api-access-tclrh\" (UniqueName: \"kubernetes.io/projected/7f2a1284-27cc-4116-b797-f9001308d2eb-kube-api-access-tclrh\") pod \"7f2a1284-27cc-4116-b797-f9001308d2eb\" (UID: \"7f2a1284-27cc-4116-b797-f9001308d2eb\") " Oct 30 12:54:27.203422 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.203384 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f2a1284-27cc-4116-b797-f9001308d2eb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f2a1284-27cc-4116-b797-f9001308d2eb-routing-overlay-volume\") pod \"7f2a1284-27cc-4116-b797-f9001308d2eb\" (UID: \"7f2a1284-27cc-4116-b797-f9001308d2eb\") " Oct 30 12:54:27.203519 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.203508 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f2a1284-27cc-4116-b797-f9001308d2eb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f2a1284-27cc-4116-b797-f9001308d2eb-provider-credentials\") pod \"7f2a1284-27cc-4116-b797-f9001308d2eb\" (UID: \"7f2a1284-27cc-4116-b797-f9001308d2eb\") " Oct 30 12:54:27.203561 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.203536 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f2a1284-27cc-4116-b797-f9001308d2eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f2a1284-27cc-4116-b797-f9001308d2eb-plugins-config-volume\") pod \"7f2a1284-27cc-4116-b797-f9001308d2eb\" (UID: \"7f2a1284-27cc-4116-b797-f9001308d2eb\") " Oct 30 12:54:27.203804 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.203773 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2a1284-27cc-4116-b797-f9001308d2eb-routing-overlay-volume" pod "7f2a1284-27cc-4116-b797-f9001308d2eb" (UID: "7f2a1284-27cc-4116-b797-f9001308d2eb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:54:27.203958 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.203937 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2a1284-27cc-4116-b797-f9001308d2eb-plugins-config-volume" pod "7f2a1284-27cc-4116-b797-f9001308d2eb" (UID: "7f2a1284-27cc-4116-b797-f9001308d2eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:54:27.204104 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.204089 4413 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f2a1284-27cc-4116-b797-f9001308d2eb-routing-overlay-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:54:27.204161 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.204109 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f2a1284-27cc-4116-b797-f9001308d2eb-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:54:27.205556 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.205536 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2a1284-27cc-4116-b797-f9001308d2eb-kube-api-access-tclrh" pod "7f2a1284-27cc-4116-b797-f9001308d2eb" (UID: "7f2a1284-27cc-4116-b797-f9001308d2eb"). InnerVolumeSpecName "kube-api-access-tclrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:27.205666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.205650 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2a1284-27cc-4116-b797-f9001308d2eb-provider-credentials" pod "7f2a1284-27cc-4116-b797-f9001308d2eb" (UID: "7f2a1284-27cc-4116-b797-f9001308d2eb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:27.305468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.305394 4413 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f2a1284-27cc-4116-b797-f9001308d2eb-provider-credentials\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:54:27.305468 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.305419 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tclrh\" (UniqueName: \"kubernetes.io/projected/7f2a1284-27cc-4116-b797-f9001308d2eb-kube-api-access-tclrh\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:54:27.865723 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.865685 4413 generic.go:356] "Generic (PLEG): container finished" podID="7f2a1284-27cc-4116-b797-f9001308d2eb" containerID="295785e16ac17f578fbb90ad5f8821660280596b50afd3a8a04509cbea1f23ee" exitCode=0 Oct 30 12:54:27.866211 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.865733 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" Oct 30 12:54:27.866211 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.865756 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" event={"ID":"7f2a1284-27cc-4116-b797-f9001308d2eb","Type":"ContainerDied","Data":"295785e16ac17f578fbb90ad5f8821660280596b50afd3a8a04509cbea1f23ee"} Oct 30 12:54:27.866211 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.865844 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5" event={"ID":"7f2a1284-27cc-4116-b797-f9001308d2eb","Type":"ContainerDied","Data":"cdd15a50176a9dc8a37092004dfca32419014af8e829b4e592021400151de71c"} Oct 30 12:54:27.866211 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.865872 4413 scope.go:122] "RemoveContainer" containerID="295785e16ac17f578fbb90ad5f8821660280596b50afd3a8a04509cbea1f23ee" Oct 30 12:54:27.877511 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.877488 4413 scope.go:122] "RemoveContainer" containerID="295785e16ac17f578fbb90ad5f8821660280596b50afd3a8a04509cbea1f23ee" Oct 30 12:54:27.877865 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:54:27.877844 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"295785e16ac17f578fbb90ad5f8821660280596b50afd3a8a04509cbea1f23ee\": container with ID starting with 295785e16ac17f578fbb90ad5f8821660280596b50afd3a8a04509cbea1f23ee not found: ID does not exist" containerID="295785e16ac17f578fbb90ad5f8821660280596b50afd3a8a04509cbea1f23ee" Oct 30 12:54:27.877983 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.877872 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"295785e16ac17f578fbb90ad5f8821660280596b50afd3a8a04509cbea1f23ee"} err="failed to get container status \"295785e16ac17f578fbb90ad5f8821660280596b50afd3a8a04509cbea1f23ee\": rpc error: code = NotFound desc = could not find container \"295785e16ac17f578fbb90ad5f8821660280596b50afd3a8a04509cbea1f23ee\": container with ID starting with 295785e16ac17f578fbb90ad5f8821660280596b50afd3a8a04509cbea1f23ee not found: ID does not exist" Oct 30 12:54:27.920439 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.920404 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5"] Oct 30 12:54:27.932023 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:27.931991 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-994c6m6t5"] Oct 30 12:54:28.484160 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:28.484127 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:29.022761 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:29.022732 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f2a1284-27cc-4116-b797-f9001308d2eb" path="/var/lib/kubelet/pods/7f2a1284-27cc-4116-b797-f9001308d2eb/volumes" Oct 30 12:54:30.181601 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:30.181562 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:31.461471 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:31.461435 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz"] Oct 30 12:54:31.551459 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:31.551430 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:31.705604 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:31.705571 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:32.031328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:32.031291 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:32.192863 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:32.192830 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:32.588397 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:32.588353 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:33.886123 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:33.886089 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:35.555217 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:35.555177 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:35.815318 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:54:35.815221 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:54:36.462494 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.462433 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" podUID="76533887-6247-4a26-9bde-650e13659209" containerName="payload-processing" containerID="cri-o://3ded870740ec088dc2f4c751f1b217303dcbb94f938761f801627e46066631c2" gracePeriod=25 Oct 30 12:54:36.739469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.739435 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" Oct 30 12:54:36.865615 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.865580 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76533887-6247-4a26-9bde-650e13659209-kube-api-access-2gnwz\" (UniqueName: \"kubernetes.io/projected/76533887-6247-4a26-9bde-650e13659209-kube-api-access-2gnwz\") pod \"76533887-6247-4a26-9bde-650e13659209\" (UID: \"76533887-6247-4a26-9bde-650e13659209\") " Oct 30 12:54:36.865799 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.865701 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76533887-6247-4a26-9bde-650e13659209-provider-credentials\" (UniqueName: \"kubernetes.io/projected/76533887-6247-4a26-9bde-650e13659209-provider-credentials\") pod \"76533887-6247-4a26-9bde-650e13659209\" (UID: \"76533887-6247-4a26-9bde-650e13659209\") " Oct 30 12:54:36.865844 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.865798 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76533887-6247-4a26-9bde-650e13659209-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76533887-6247-4a26-9bde-650e13659209-routing-overlay-volume\") pod \"76533887-6247-4a26-9bde-650e13659209\" (UID: \"76533887-6247-4a26-9bde-650e13659209\") " Oct 30 12:54:36.865899 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.865880 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76533887-6247-4a26-9bde-650e13659209-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76533887-6247-4a26-9bde-650e13659209-plugins-config-volume\") pod \"76533887-6247-4a26-9bde-650e13659209\" (UID: \"76533887-6247-4a26-9bde-650e13659209\") " Oct 30 12:54:36.866186 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.866160 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76533887-6247-4a26-9bde-650e13659209-routing-overlay-volume" pod "76533887-6247-4a26-9bde-650e13659209" (UID: "76533887-6247-4a26-9bde-650e13659209"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:54:36.866336 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.866231 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76533887-6247-4a26-9bde-650e13659209-plugins-config-volume" pod "76533887-6247-4a26-9bde-650e13659209" (UID: "76533887-6247-4a26-9bde-650e13659209"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:54:36.866510 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.866492 4413 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76533887-6247-4a26-9bde-650e13659209-routing-overlay-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:54:36.866566 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.866516 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76533887-6247-4a26-9bde-650e13659209-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:54:36.868003 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.867984 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76533887-6247-4a26-9bde-650e13659209-kube-api-access-2gnwz" pod "76533887-6247-4a26-9bde-650e13659209" (UID: "76533887-6247-4a26-9bde-650e13659209"). InnerVolumeSpecName "kube-api-access-2gnwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:36.868099 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.868000 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76533887-6247-4a26-9bde-650e13659209-provider-credentials" pod "76533887-6247-4a26-9bde-650e13659209" (UID: "76533887-6247-4a26-9bde-650e13659209"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:54:36.949975 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.949940 4413 generic.go:356] "Generic (PLEG): container finished" podID="76533887-6247-4a26-9bde-650e13659209" containerID="3ded870740ec088dc2f4c751f1b217303dcbb94f938761f801627e46066631c2" exitCode=0 Oct 30 12:54:36.950150 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.949981 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" Oct 30 12:54:36.950150 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.950020 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" event={"ID":"76533887-6247-4a26-9bde-650e13659209","Type":"ContainerDied","Data":"3ded870740ec088dc2f4c751f1b217303dcbb94f938761f801627e46066631c2"} Oct 30 12:54:36.950150 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.950069 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz" event={"ID":"76533887-6247-4a26-9bde-650e13659209","Type":"ContainerDied","Data":"23879a166b3c066b7104dc52d6c41e940a4d09a5af60929c1d2806204d835e6b"} Oct 30 12:54:36.950150 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.950093 4413 scope.go:122] "RemoveContainer" containerID="3ded870740ec088dc2f4c751f1b217303dcbb94f938761f801627e46066631c2" Oct 30 12:54:36.961834 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.961814 4413 scope.go:122] "RemoveContainer" containerID="3ded870740ec088dc2f4c751f1b217303dcbb94f938761f801627e46066631c2" Oct 30 12:54:36.962121 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:54:36.962101 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ded870740ec088dc2f4c751f1b217303dcbb94f938761f801627e46066631c2\": container with ID starting with 3ded870740ec088dc2f4c751f1b217303dcbb94f938761f801627e46066631c2 not found: ID does not exist" containerID="3ded870740ec088dc2f4c751f1b217303dcbb94f938761f801627e46066631c2" Oct 30 12:54:36.962169 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.962133 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ded870740ec088dc2f4c751f1b217303dcbb94f938761f801627e46066631c2"} err="failed to get container status \"3ded870740ec088dc2f4c751f1b217303dcbb94f938761f801627e46066631c2\": rpc error: code = NotFound desc = could not find container \"3ded870740ec088dc2f4c751f1b217303dcbb94f938761f801627e46066631c2\": container with ID starting with 3ded870740ec088dc2f4c751f1b217303dcbb94f938761f801627e46066631c2 not found: ID does not exist" Oct 30 12:54:36.967582 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.967538 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gnwz\" (UniqueName: \"kubernetes.io/projected/76533887-6247-4a26-9bde-650e13659209-kube-api-access-2gnwz\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:54:36.967582 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:36.967559 4413 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76533887-6247-4a26-9bde-650e13659209-provider-credentials\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:54:37.004849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.004824 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz"] Oct 30 12:54:37.021592 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.021567 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-12fd33/payload-processing-external-model-e2e-extmod-b-12fd33-d76f5lkfz"] Oct 30 12:54:37.251500 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.251424 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7"] Oct 30 12:54:37.252233 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.252219 4413 cpu_manager.go:414] "containerMap: removing container" podUID="76533887-6247-4a26-9bde-650e13659209" containerName="payload-processing" Oct 30 12:54:37.252317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.252234 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76533887-6247-4a26-9bde-650e13659209" containerName="payload-processing" Oct 30 12:54:37.252317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.252298 4413 cpu_manager.go:414] "containerMap: removing container" podUID="7f2a1284-27cc-4116-b797-f9001308d2eb" containerName="payload-processing" Oct 30 12:54:37.252317 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.252307 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f2a1284-27cc-4116-b797-f9001308d2eb" containerName="payload-processing" Oct 30 12:54:37.252439 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.252344 4413 cpu_manager.go:414] "containerMap: removing container" podUID="13f9821e-d4d1-4942-b2f4-0580f8ca9728" containerName="payload-processing" Oct 30 12:54:37.252439 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.252351 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f9821e-d4d1-4942-b2f4-0580f8ca9728" containerName="payload-processing" Oct 30 12:54:37.252523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.252513 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f2a1284-27cc-4116-b797-f9001308d2eb" containerName="payload-processing" Oct 30 12:54:37.252562 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.252528 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f9821e-d4d1-4942-b2f4-0580f8ca9728" containerName="payload-processing" Oct 30 12:54:37.252562 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.252541 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="76533887-6247-4a26-9bde-650e13659209" containerName="payload-processing" Oct 30 12:54:37.257901 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.257884 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 12:54:37.274999 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.274973 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-eab799-serving-cert\"" Oct 30 12:54:37.294319 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.294289 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7"] Oct 30 12:54:37.372236 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.372192 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2d9a96-1d2e-4eee-946a-d26618a955ca-metrics-certs\") pod \"maas-api-e2e-inf-a-eab799-7f655c44db-kczd7\" (UID: \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 12:54:37.372522 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.372501 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd2d9a96-1d2e-4eee-946a-d26618a955ca-maas-api-tls\") pod \"maas-api-e2e-inf-a-eab799-7f655c44db-kczd7\" (UID: \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 12:54:37.372668 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.372637 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66bcj\" (UniqueName: \"kubernetes.io/projected/fd2d9a96-1d2e-4eee-946a-d26618a955ca-kube-api-access-66bcj\") pod \"maas-api-e2e-inf-a-eab799-7f655c44db-kczd7\" (UID: \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 12:54:37.372741 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.372717 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2d9a96-1d2e-4eee-946a-d26618a955ca-openshift-service-ca\") pod \"maas-api-e2e-inf-a-eab799-7f655c44db-kczd7\" (UID: \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 12:54:37.414612 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.414578 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:37.474957 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.474900 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd2d9a96-1d2e-4eee-946a-d26618a955ca-maas-api-tls\") pod \"maas-api-e2e-inf-a-eab799-7f655c44db-kczd7\" (UID: \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 12:54:37.475175 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.475023 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66bcj\" (UniqueName: \"kubernetes.io/projected/fd2d9a96-1d2e-4eee-946a-d26618a955ca-kube-api-access-66bcj\") pod \"maas-api-e2e-inf-a-eab799-7f655c44db-kczd7\" (UID: \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 12:54:37.475175 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.475063 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2d9a96-1d2e-4eee-946a-d26618a955ca-openshift-service-ca\") pod \"maas-api-e2e-inf-a-eab799-7f655c44db-kczd7\" (UID: \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 12:54:37.475351 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.475244 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2d9a96-1d2e-4eee-946a-d26618a955ca-metrics-certs\") pod \"maas-api-e2e-inf-a-eab799-7f655c44db-kczd7\" (UID: \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 12:54:37.475608 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:54:37.475583 4413 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-serving-cert: secret "maas-api-e2e-inf-a-eab799-serving-cert" not found Oct 30 12:54:37.475853 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:54:37.475826 4413 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd2d9a96-1d2e-4eee-946a-d26618a955ca-maas-api-tls podName:fd2d9a96-1d2e-4eee-946a-d26618a955ca nodeName:}" failed. No retries permitted until 2026-10-30 12:54:37.975804114 +0000 UTC m=+3631.085646974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd2d9a96-1d2e-4eee-946a-d26618a955ca-maas-api-tls") pod "maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" (UID: "fd2d9a96-1d2e-4eee-946a-d26618a955ca") : secret "maas-api-e2e-inf-a-eab799-serving-cert" not found Oct 30 12:54:37.476511 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.476471 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2d9a96-1d2e-4eee-946a-d26618a955ca-openshift-service-ca\") pod \"maas-api-e2e-inf-a-eab799-7f655c44db-kczd7\" (UID: \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 12:54:37.507981 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.507895 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2d9a96-1d2e-4eee-946a-d26618a955ca-metrics-certs\") pod \"maas-api-e2e-inf-a-eab799-7f655c44db-kczd7\" (UID: \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 12:54:37.533099 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.533059 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66bcj\" (UniqueName: \"kubernetes.io/projected/fd2d9a96-1d2e-4eee-946a-d26618a955ca-kube-api-access-66bcj\") pod \"maas-api-e2e-inf-a-eab799-7f655c44db-kczd7\" (UID: \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 12:54:37.554325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.549035 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:37.744462 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.744405 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:37.992646 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.992616 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd2d9a96-1d2e-4eee-946a-d26618a955ca-maas-api-tls\") pod \"maas-api-e2e-inf-a-eab799-7f655c44db-kczd7\" (UID: \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 12:54:37.995224 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:37.995192 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd2d9a96-1d2e-4eee-946a-d26618a955ca-maas-api-tls\") pod \"maas-api-e2e-inf-a-eab799-7f655c44db-kczd7\" (UID: \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 12:54:38.172815 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:38.172782 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 12:54:38.529671 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:38.529641 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7"] Oct 30 12:54:38.536282 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:38.535866 4413 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 12:54:38.967390 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:38.967359 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" event={"ID":"fd2d9a96-1d2e-4eee-946a-d26618a955ca","Type":"ContainerStarted","Data":"2a69b26e2c67d8d62163be3a56178e1739059c9f9dc87e9473da75b307b6c9dc"} Oct 30 12:54:39.032079 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:39.032048 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76533887-6247-4a26-9bde-650e13659209" path="/var/lib/kubelet/pods/76533887-6247-4a26-9bde-650e13659209/volumes" Oct 30 12:54:39.574126 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:39.574087 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:39.977206 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:39.977174 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" event={"ID":"fd2d9a96-1d2e-4eee-946a-d26618a955ca","Type":"ContainerStarted","Data":"894357e69d40d4f2651a451f8dc0c12aef8971fea1fc17c8fd5ccde859ea7d77"} Oct 30 12:54:39.977712 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:39.977290 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 12:54:40.021662 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:40.021591 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" podStartSLOduration=2.456556231 podStartE2EDuration="3.021572308s" totalImagesPullingTime="565.016077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:40.019293127 +0000 UTC m=+3633.129136008" watchObservedRunningTime="2026-10-30 12:54:40.021572308 +0000 UTC m=+3633.131415190" Oct 30 12:54:40.717227 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:40.717181 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:40.886718 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:40.886681 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:41.113953 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:41.113846 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:41.235017 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:41.234972 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:41.365800 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:41.365712 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:41.705703 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:41.705665 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:41.842113 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:41.842076 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:41.980717 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:41.980625 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:42.097438 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:42.097409 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:42.237350 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:42.237233 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:43.407931 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:43.407899 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:45.242199 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:45.242163 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:45.991988 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:45.991963 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 12:54:46.003701 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:46.003660 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:46.156938 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:46.156908 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:47.067469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:47.067432 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:47.734928 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:47.734875 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:47.929656 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:47.929614 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:49.011907 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:49.011830 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:50.041188 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:50.041155 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:50.158835 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:50.158805 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:50.502998 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:50.502945 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:50.755276 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:50.755114 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:51.021273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:51.021167 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:51.189645 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:51.189601 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:52.519119 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:52.519080 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:54.373378 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:54.373333 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:55.063118 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:55.063085 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:55.190439 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:55.190410 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:56.190516 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:56.190481 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:56.804562 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:56.804531 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:56.945045 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:56.945016 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:54:58.161507 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:54:58.161475 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:00.054364 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:00.054331 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:01.508565 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:01.508532 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:01.965185 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:01.965147 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:03.577904 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:03.577871 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:05.143089 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:05.143056 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:05.818335 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:55:05.818297 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:55:06.781193 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:06.781153 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:08.461286 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:08.461238 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:10.094302 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:10.094240 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:11.774134 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:11.774097 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:13.450456 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:13.450424 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:15.256804 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:15.256766 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:16.843580 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:16.843544 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:18.698923 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:18.698877 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:20.523126 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:20.523088 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:22.276665 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:22.276634 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:23.835616 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:23.835586 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:25.450242 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:25.450209 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:25.580913 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:25.580873 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:27.432618 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:27.432586 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:27.734719 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:27.734637 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:28.233455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:28.233428 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:28.481816 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:28.481782 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:28.607651 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:28.607569 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:29.014769 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:29.014737 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:29.629984 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:29.629950 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:29.829411 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:29.829371 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:30.017650 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:30.017532 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:31.486516 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:31.486486 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:32.558800 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:32.558753 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:32.715678 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:32.715638 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:33.395404 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:33.395372 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:33.512553 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:33.512505 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:33.636305 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:33.636267 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:34.193156 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:34.193129 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:34.383526 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:34.383495 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:34.551000 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:34.550932 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:35.571615 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:35.571585 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:35.816469 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:55:35.816424 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:55:36.875407 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:36.875371 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:36.967390 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:36.967363 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m"] Oct 30 12:55:36.972426 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:36.972407 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" Oct 30 12:55:36.991708 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:36.991683 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m"] Oct 30 12:55:36.993670 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:36.993648 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:36.997965 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:36.997942 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e9f184\"" Oct 30 12:55:36.998089 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:36.997960 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e9f184-dockercfg-djfwr\"" Oct 30 12:55:37.035367 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.035338 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7"] Oct 30 12:55:37.040148 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.040126 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" Oct 30 12:55:37.044090 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.044070 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfcmq\" (UniqueName: \"kubernetes.io/projected/fe249b4d-be8a-4135-acfc-568903715ef6-kube-api-access-sfcmq\") pod \"payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m\" (UID: \"fe249b4d-be8a-4135-acfc-568903715ef6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" Oct 30 12:55:37.044300 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.044244 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe249b4d-be8a-4135-acfc-568903715ef6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m\" (UID: \"fe249b4d-be8a-4135-acfc-568903715ef6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" Oct 30 12:55:37.057415 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.057388 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7"] Oct 30 12:55:37.159330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.153497 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg2lj\" (UniqueName: \"kubernetes.io/projected/ed73650f-c307-4db1-80c1-9b129dd488c4-kube-api-access-wg2lj\") pod \"payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7\" (UID: \"ed73650f-c307-4db1-80c1-9b129dd488c4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" Oct 30 12:55:37.159330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.153568 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfcmq\" (UniqueName: \"kubernetes.io/projected/fe249b4d-be8a-4135-acfc-568903715ef6-kube-api-access-sfcmq\") pod \"payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m\" (UID: \"fe249b4d-be8a-4135-acfc-568903715ef6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" Oct 30 12:55:37.159330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.153724 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe249b4d-be8a-4135-acfc-568903715ef6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m\" (UID: \"fe249b4d-be8a-4135-acfc-568903715ef6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" Oct 30 12:55:37.159330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.153888 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed73650f-c307-4db1-80c1-9b129dd488c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7\" (UID: \"ed73650f-c307-4db1-80c1-9b129dd488c4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" Oct 30 12:55:37.159330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.155781 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe249b4d-be8a-4135-acfc-568903715ef6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m\" (UID: \"fe249b4d-be8a-4135-acfc-568903715ef6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" Oct 30 12:55:37.223976 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.223942 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfcmq\" (UniqueName: \"kubernetes.io/projected/fe249b4d-be8a-4135-acfc-568903715ef6-kube-api-access-sfcmq\") pod \"payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m\" (UID: \"fe249b4d-be8a-4135-acfc-568903715ef6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" Oct 30 12:55:37.255564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.255527 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg2lj\" (UniqueName: \"kubernetes.io/projected/ed73650f-c307-4db1-80c1-9b129dd488c4-kube-api-access-wg2lj\") pod \"payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7\" (UID: \"ed73650f-c307-4db1-80c1-9b129dd488c4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" Oct 30 12:55:37.255759 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.255737 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed73650f-c307-4db1-80c1-9b129dd488c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7\" (UID: \"ed73650f-c307-4db1-80c1-9b129dd488c4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" Oct 30 12:55:37.256489 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.256467 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed73650f-c307-4db1-80c1-9b129dd488c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7\" (UID: \"ed73650f-c307-4db1-80c1-9b129dd488c4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" Oct 30 12:55:37.279267 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.279229 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg2lj\" (UniqueName: \"kubernetes.io/projected/ed73650f-c307-4db1-80c1-9b129dd488c4-kube-api-access-wg2lj\") pod \"payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7\" (UID: \"ed73650f-c307-4db1-80c1-9b129dd488c4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" Oct 30 12:55:37.287745 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.287710 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" Oct 30 12:55:37.367340 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.367306 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" Oct 30 12:55:37.535156 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.535127 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:37.591938 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.591887 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m"] Oct 30 12:55:37.683395 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:37.683046 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7"] Oct 30 12:55:38.573011 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:38.572977 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" event={"ID":"fe249b4d-be8a-4135-acfc-568903715ef6","Type":"ContainerStarted","Data":"e25e452e6225db4a7b91f13763d2716e412a6bcd57f3abe66605dccddcdbe2c6"} Oct 30 12:55:38.573523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:38.573038 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" event={"ID":"fe249b4d-be8a-4135-acfc-568903715ef6","Type":"ContainerStarted","Data":"74940f8fa45f40af8a1b7e065e635ecd37c648bf5bc8a6e0ef278bf32a62aacd"} Oct 30 12:55:38.573523 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:38.573070 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" Oct 30 12:55:38.574806 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:38.574777 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" event={"ID":"ed73650f-c307-4db1-80c1-9b129dd488c4","Type":"ContainerStarted","Data":"48ade5fc144d34274f82e39b54d8a78aade6d47cdd6339a946beb984ff3dbcee"} Oct 30 12:55:38.574937 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:38.574821 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" event={"ID":"ed73650f-c307-4db1-80c1-9b129dd488c4","Type":"ContainerStarted","Data":"8203ac9fbf82c0259ab7ecf0ea096a3d2541902fc0c4e6b5f0e41f7e59ad45c1"} Oct 30 12:55:38.574937 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:38.574874 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" Oct 30 12:55:38.614833 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:38.614791 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" podStartSLOduration=2.614778087 podStartE2EDuration="2.614778087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:38.614483751 +0000 UTC m=+3691.724326657" watchObservedRunningTime="2026-10-30 12:55:38.614778087 +0000 UTC m=+3691.724620967" Oct 30 12:55:38.664672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:38.664631 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" podStartSLOduration=1.664617195 podStartE2EDuration="1.664617195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:38.659816272 +0000 UTC m=+3691.769659336" watchObservedRunningTime="2026-10-30 12:55:38.664617195 +0000 UTC m=+3691.774460076" Oct 30 12:55:39.198436 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:39.198404 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:39.318020 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:39.317991 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl"] Oct 30 12:55:40.892004 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:40.891958 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:42.652231 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:42.652187 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:44.205845 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.205814 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj"] Oct 30 12:55:44.210573 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.210553 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" Oct 30 12:55:44.230045 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.230021 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj"] Oct 30 12:55:44.232803 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.232784 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6b67f8-serving-cert\"" Oct 30 12:55:44.314393 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.314361 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:44.319151 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.319126 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" podUID="56af769c-6f94-44de-ad72-b0e406e531a8" containerName="payload-processing" containerID="cri-o://6d9163a713ac0a508a0ed3a3d0b211a265ed0eae964d6b21fa0ab90852eafba3" gracePeriod=25 Oct 30 12:55:44.356857 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.356832 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea820a52-0cbd-4fc8-9539-4aa9810e1836-metrics-certs\") pod \"maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj\" (UID: \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" Oct 30 12:55:44.357074 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.357051 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea820a52-0cbd-4fc8-9539-4aa9810e1836-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj\" (UID: \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" Oct 30 12:55:44.357163 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.357150 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea820a52-0cbd-4fc8-9539-4aa9810e1836-maas-api-tls\") pod \"maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj\" (UID: \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" Oct 30 12:55:44.357217 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.357204 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7wj9\" (UniqueName: \"kubernetes.io/projected/ea820a52-0cbd-4fc8-9539-4aa9810e1836-kube-api-access-n7wj9\") pod \"maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj\" (UID: \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" Oct 30 12:55:44.458677 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.458575 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea820a52-0cbd-4fc8-9539-4aa9810e1836-metrics-certs\") pod \"maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj\" (UID: \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" Oct 30 12:55:44.458935 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.458794 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:44.458935 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.458824 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea820a52-0cbd-4fc8-9539-4aa9810e1836-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj\" (UID: \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" Oct 30 12:55:44.458935 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.458918 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea820a52-0cbd-4fc8-9539-4aa9810e1836-maas-api-tls\") pod \"maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj\" (UID: \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" Oct 30 12:55:44.459146 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.458965 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7wj9\" (UniqueName: \"kubernetes.io/projected/ea820a52-0cbd-4fc8-9539-4aa9810e1836-kube-api-access-n7wj9\") pod \"maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj\" (UID: \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" Oct 30 12:55:44.459499 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.459475 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea820a52-0cbd-4fc8-9539-4aa9810e1836-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj\" (UID: \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" Oct 30 12:55:44.461455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.461430 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea820a52-0cbd-4fc8-9539-4aa9810e1836-metrics-certs\") pod \"maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj\" (UID: \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" Oct 30 12:55:44.461455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.461453 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea820a52-0cbd-4fc8-9539-4aa9810e1836-maas-api-tls\") pod \"maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj\" (UID: \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" Oct 30 12:55:44.483640 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.483607 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7wj9\" (UniqueName: \"kubernetes.io/projected/ea820a52-0cbd-4fc8-9539-4aa9810e1836-kube-api-access-n7wj9\") pod \"maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj\" (UID: \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" Oct 30 12:55:44.528292 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.527563 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" Oct 30 12:55:44.584968 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.584941 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" Oct 30 12:55:44.628414 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.628359 4413 generic.go:356] "Generic (PLEG): container finished" podID="56af769c-6f94-44de-ad72-b0e406e531a8" containerID="6d9163a713ac0a508a0ed3a3d0b211a265ed0eae964d6b21fa0ab90852eafba3" exitCode=0 Oct 30 12:55:44.628629 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.628598 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" event={"ID":"56af769c-6f94-44de-ad72-b0e406e531a8","Type":"ContainerDied","Data":"6d9163a713ac0a508a0ed3a3d0b211a265ed0eae964d6b21fa0ab90852eafba3"} Oct 30 12:55:44.628692 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.628657 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" event={"ID":"56af769c-6f94-44de-ad72-b0e406e531a8","Type":"ContainerDied","Data":"51df789ff32aa4ef8e842237cbe74a27231dfcfa11a00a51fbffbe0078ea3361"} Oct 30 12:55:44.628692 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.628679 4413 scope.go:122] "RemoveContainer" containerID="6d9163a713ac0a508a0ed3a3d0b211a265ed0eae964d6b21fa0ab90852eafba3" Oct 30 12:55:44.628837 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.628812 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl" Oct 30 12:55:44.648520 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.648493 4413 scope.go:122] "RemoveContainer" containerID="6d9163a713ac0a508a0ed3a3d0b211a265ed0eae964d6b21fa0ab90852eafba3" Oct 30 12:55:44.650765 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:55:44.650720 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9163a713ac0a508a0ed3a3d0b211a265ed0eae964d6b21fa0ab90852eafba3\": container with ID starting with 6d9163a713ac0a508a0ed3a3d0b211a265ed0eae964d6b21fa0ab90852eafba3 not found: ID does not exist" containerID="6d9163a713ac0a508a0ed3a3d0b211a265ed0eae964d6b21fa0ab90852eafba3" Oct 30 12:55:44.650923 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.650778 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9163a713ac0a508a0ed3a3d0b211a265ed0eae964d6b21fa0ab90852eafba3"} err="failed to get container status \"6d9163a713ac0a508a0ed3a3d0b211a265ed0eae964d6b21fa0ab90852eafba3\": rpc error: code = NotFound desc = could not find container \"6d9163a713ac0a508a0ed3a3d0b211a265ed0eae964d6b21fa0ab90852eafba3\": container with ID starting with 6d9163a713ac0a508a0ed3a3d0b211a265ed0eae964d6b21fa0ab90852eafba3 not found: ID does not exist" Oct 30 12:55:44.661722 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.661679 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56af769c-6f94-44de-ad72-b0e406e531a8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/56af769c-6f94-44de-ad72-b0e406e531a8-provider-credentials\") pod \"56af769c-6f94-44de-ad72-b0e406e531a8\" (UID: \"56af769c-6f94-44de-ad72-b0e406e531a8\") " Oct 30 12:55:44.661911 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.661739 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56af769c-6f94-44de-ad72-b0e406e531a8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56af769c-6f94-44de-ad72-b0e406e531a8-routing-overlay-volume\") pod \"56af769c-6f94-44de-ad72-b0e406e531a8\" (UID: \"56af769c-6f94-44de-ad72-b0e406e531a8\") " Oct 30 12:55:44.661911 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.661846 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56af769c-6f94-44de-ad72-b0e406e531a8-kube-api-access-scdt4\" (UniqueName: \"kubernetes.io/projected/56af769c-6f94-44de-ad72-b0e406e531a8-kube-api-access-scdt4\") pod \"56af769c-6f94-44de-ad72-b0e406e531a8\" (UID: \"56af769c-6f94-44de-ad72-b0e406e531a8\") " Oct 30 12:55:44.662026 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.661935 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56af769c-6f94-44de-ad72-b0e406e531a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56af769c-6f94-44de-ad72-b0e406e531a8-plugins-config-volume\") pod \"56af769c-6f94-44de-ad72-b0e406e531a8\" (UID: \"56af769c-6f94-44de-ad72-b0e406e531a8\") " Oct 30 12:55:44.663175 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.663142 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56af769c-6f94-44de-ad72-b0e406e531a8-plugins-config-volume" pod "56af769c-6f94-44de-ad72-b0e406e531a8" (UID: "56af769c-6f94-44de-ad72-b0e406e531a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:55:44.663471 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.663445 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56af769c-6f94-44de-ad72-b0e406e531a8-routing-overlay-volume" pod "56af769c-6f94-44de-ad72-b0e406e531a8" (UID: "56af769c-6f94-44de-ad72-b0e406e531a8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 12:55:44.665661 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.665628 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56af769c-6f94-44de-ad72-b0e406e531a8-kube-api-access-scdt4" pod "56af769c-6f94-44de-ad72-b0e406e531a8" (UID: "56af769c-6f94-44de-ad72-b0e406e531a8"). InnerVolumeSpecName "kube-api-access-scdt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:55:44.665751 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.665710 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56af769c-6f94-44de-ad72-b0e406e531a8-provider-credentials" pod "56af769c-6f94-44de-ad72-b0e406e531a8" (UID: "56af769c-6f94-44de-ad72-b0e406e531a8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 12:55:44.764430 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.764357 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56af769c-6f94-44de-ad72-b0e406e531a8-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:55:44.764430 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.764379 4413 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56af769c-6f94-44de-ad72-b0e406e531a8-provider-credentials\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:55:44.764430 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.764388 4413 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56af769c-6f94-44de-ad72-b0e406e531a8-routing-overlay-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:55:44.764430 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.764396 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scdt4\" (UniqueName: \"kubernetes.io/projected/56af769c-6f94-44de-ad72-b0e406e531a8-kube-api-access-scdt4\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 12:55:44.859205 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.859173 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj"] Oct 30 12:55:44.859421 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:55:44.859398 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea820a52_0cbd_4fc8_9539_4aa9810e1836.slice/crio-b8e1b7bda5ea68a8363afc3f7165603fe50d9a5d22b7dcc5619a17b1c4d2b9ae WatchSource:0}: Error finding container b8e1b7bda5ea68a8363afc3f7165603fe50d9a5d22b7dcc5619a17b1c4d2b9ae: Status 404 returned error can't find the container with id b8e1b7bda5ea68a8363afc3f7165603fe50d9a5d22b7dcc5619a17b1c4d2b9ae Oct 30 12:55:44.976462 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.976439 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl"] Oct 30 12:55:44.980504 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:44.980481 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c9884c959-7qlvl"] Oct 30 12:55:45.021740 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:45.021682 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56af769c-6f94-44de-ad72-b0e406e531a8" path="/var/lib/kubelet/pods/56af769c-6f94-44de-ad72-b0e406e531a8/volumes" Oct 30 12:55:45.640856 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:45.640818 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" event={"ID":"ea820a52-0cbd-4fc8-9539-4aa9810e1836","Type":"ContainerStarted","Data":"19005136cebf590cbb4fc84b1e0f28c200a5252f9977e334a80c0d630e737fb9"} Oct 30 12:55:45.641376 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:45.640896 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" Oct 30 12:55:45.641376 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:45.640923 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" event={"ID":"ea820a52-0cbd-4fc8-9539-4aa9810e1836","Type":"ContainerStarted","Data":"b8e1b7bda5ea68a8363afc3f7165603fe50d9a5d22b7dcc5619a17b1c4d2b9ae"} Oct 30 12:55:45.683080 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:45.683032 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" podStartSLOduration=1.353735411 podStartE2EDuration="1.683019043s" totalImagesPullingTime="329.283632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:45.682347303 +0000 UTC m=+3698.792190185" watchObservedRunningTime="2026-10-30 12:55:45.683019043 +0000 UTC m=+3698.792861923" Oct 30 12:55:45.724810 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:45.724772 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:45.838520 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:45.838487 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:46.068330 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:46.068228 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:47.846874 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:47.846837 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:49.429053 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:49.429025 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:49.586579 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:49.586555 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" Oct 30 12:55:49.586774 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:49.586616 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" Oct 30 12:55:51.090089 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:51.090059 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:51.655231 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:51.655200 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" Oct 30 12:55:52.820867 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:52.820831 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:54.450049 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:54.450018 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:56.296827 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:56.296788 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:58.263297 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:58.263268 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:55:59.948177 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:55:59.948142 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:01.709914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:01.709864 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:01.838037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:01.837987 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:03.567399 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:03.566823 4413 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 12:56:03.567399 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:03.566922 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9x4ps" podUID="b2d21fd6-1a62-4b69-af88-d5038a533f90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 12:56:03.967277 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:03.966344 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:05.639554 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:05.639521 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:05.814903 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:56:05.814867 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:56:07.375498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:07.375464 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:09.057617 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:09.057586 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:10.742479 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:10.742440 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:12.404721 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:12.404670 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:14.223443 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:14.223409 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:15.939497 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:15.939465 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:17.616418 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:17.616390 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:19.276273 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:19.276171 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:20.951419 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:20.951387 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:22.557653 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:22.557603 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:22.707849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:22.705959 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:24.501828 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:24.501799 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:25.999607 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:25.999577 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:26.036673 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.036648 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt"] Oct 30 12:56:26.037478 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.037463 4413 cpu_manager.go:414] "containerMap: removing container" podUID="56af769c-6f94-44de-ad72-b0e406e531a8" containerName="payload-processing" Oct 30 12:56:26.037532 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.037480 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56af769c-6f94-44de-ad72-b0e406e531a8" containerName="payload-processing" Oct 30 12:56:26.037635 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.037626 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="56af769c-6f94-44de-ad72-b0e406e531a8" containerName="payload-processing" Oct 30 12:56:26.041856 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.041841 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" Oct 30 12:56:26.060957 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.060927 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt"] Oct 30 12:56:26.067994 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.067921 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e9f184-serving-cert\"" Oct 30 12:56:26.081148 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.081068 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d9217c-a901-46fa-ba61-21bc01e658cb-maas-api-tls\") pod \"maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt\" (UID: \"22d9217c-a901-46fa-ba61-21bc01e658cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" Oct 30 12:56:26.081427 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.081402 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22d9217c-a901-46fa-ba61-21bc01e658cb-metrics-certs\") pod \"maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt\" (UID: \"22d9217c-a901-46fa-ba61-21bc01e658cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" Oct 30 12:56:26.081658 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.081630 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d9217c-a901-46fa-ba61-21bc01e658cb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt\" (UID: \"22d9217c-a901-46fa-ba61-21bc01e658cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" Oct 30 12:56:26.081761 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.081691 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h947z\" (UniqueName: \"kubernetes.io/projected/22d9217c-a901-46fa-ba61-21bc01e658cb-kube-api-access-h947z\") pod \"maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt\" (UID: \"22d9217c-a901-46fa-ba61-21bc01e658cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" Oct 30 12:56:26.120564 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.120534 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:26.182537 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.182514 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d9217c-a901-46fa-ba61-21bc01e658cb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt\" (UID: \"22d9217c-a901-46fa-ba61-21bc01e658cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" Oct 30 12:56:26.182666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.182557 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h947z\" (UniqueName: \"kubernetes.io/projected/22d9217c-a901-46fa-ba61-21bc01e658cb-kube-api-access-h947z\") pod \"maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt\" (UID: \"22d9217c-a901-46fa-ba61-21bc01e658cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" Oct 30 12:56:26.182666 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.182631 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d9217c-a901-46fa-ba61-21bc01e658cb-maas-api-tls\") pod \"maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt\" (UID: \"22d9217c-a901-46fa-ba61-21bc01e658cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" Oct 30 12:56:26.182746 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.182725 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22d9217c-a901-46fa-ba61-21bc01e658cb-metrics-certs\") pod \"maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt\" (UID: \"22d9217c-a901-46fa-ba61-21bc01e658cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" Oct 30 12:56:26.183157 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.183137 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d9217c-a901-46fa-ba61-21bc01e658cb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt\" (UID: \"22d9217c-a901-46fa-ba61-21bc01e658cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" Oct 30 12:56:26.185282 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.185226 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d9217c-a901-46fa-ba61-21bc01e658cb-maas-api-tls\") pod \"maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt\" (UID: \"22d9217c-a901-46fa-ba61-21bc01e658cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" Oct 30 12:56:26.185372 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.185311 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22d9217c-a901-46fa-ba61-21bc01e658cb-metrics-certs\") pod \"maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt\" (UID: \"22d9217c-a901-46fa-ba61-21bc01e658cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" Oct 30 12:56:26.206180 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.206156 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h947z\" (UniqueName: \"kubernetes.io/projected/22d9217c-a901-46fa-ba61-21bc01e658cb-kube-api-access-h947z\") pod \"maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt\" (UID: \"22d9217c-a901-46fa-ba61-21bc01e658cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" Oct 30 12:56:26.244915 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.244890 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:26.356780 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.356706 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" Oct 30 12:56:26.382849 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.382811 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:26.674712 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.674681 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt"] Oct 30 12:56:26.675994 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:56:26.675969 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d9217c_a901_46fa_ba61_21bc01e658cb.slice/crio-19f68d5dc0f2bc1b38ba7f16af6ec2d5e0f5de2db1b6b6be5ca4c9ddc6631377 WatchSource:0}: Error finding container 19f68d5dc0f2bc1b38ba7f16af6ec2d5e0f5de2db1b6b6be5ca4c9ddc6631377: Status 404 returned error can't find the container with id 19f68d5dc0f2bc1b38ba7f16af6ec2d5e0f5de2db1b6b6be5ca4c9ddc6631377 Oct 30 12:56:26.790898 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:26.790868 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:27.139797 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:27.139758 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" event={"ID":"22d9217c-a901-46fa-ba61-21bc01e658cb","Type":"ContainerStarted","Data":"19f68d5dc0f2bc1b38ba7f16af6ec2d5e0f5de2db1b6b6be5ca4c9ddc6631377"} Oct 30 12:56:27.723209 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:27.723175 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:27.852633 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:27.852606 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:28.158230 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:28.158190 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" event={"ID":"22d9217c-a901-46fa-ba61-21bc01e658cb","Type":"ContainerStarted","Data":"1af290ebfa8ddac13fa6ff344814dca42e9b79b499523279286a7035b13fbb53"} Oct 30 12:56:28.158847 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:28.158334 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" Oct 30 12:56:28.163108 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:28.163083 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:28.201622 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:28.201574 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" podStartSLOduration=1.751756826 podStartE2EDuration="2.20155854s" totalImagesPullingTime="449.801714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:28.201116684 +0000 UTC m=+3741.310959580" watchObservedRunningTime="2026-10-30 12:56:28.20155854 +0000 UTC m=+3741.311401420" Oct 30 12:56:28.302325 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:28.302238 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:28.454376 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:28.454297 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:29.080081 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:29.080052 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:29.220194 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:29.220157 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:29.855392 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:29.855338 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:30.318522 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:30.318467 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:31.526740 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:31.526707 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:32.294302 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:32.294231 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:33.328119 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:33.328090 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:33.837035 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:33.837001 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:34.181099 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:34.181071 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" Oct 30 12:56:34.196731 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:34.196703 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:34.327381 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:34.327352 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:34.524685 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:34.524474 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:35.815093 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:56:35.815054 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:56:35.851515 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:35.851482 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:37.599678 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:37.599644 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:38.317140 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:38.317075 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:38.455546 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:38.455514 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:39.423151 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:39.423117 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:40.101839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:40.101806 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:40.253305 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:40.253241 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:40.329584 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:40.329553 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj"] Oct 30 12:56:40.334675 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:40.334655 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" Oct 30 12:56:40.361910 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:40.361852 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-734ffe\"" Oct 30 12:56:40.362088 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:40.361901 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-734ffe-dockercfg-vmj47\"" Oct 30 12:56:40.366572 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:40.366551 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj"] Oct 30 12:56:40.444422 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:40.444389 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b278n\" (UniqueName: \"kubernetes.io/projected/935f35f7-28c0-4d05-ab8e-1a2042d7aab4-kube-api-access-b278n\") pod \"payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj\" (UID: \"935f35f7-28c0-4d05-ab8e-1a2042d7aab4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" Oct 30 12:56:40.444980 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:40.444956 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935f35f7-28c0-4d05-ab8e-1a2042d7aab4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj\" (UID: \"935f35f7-28c0-4d05-ab8e-1a2042d7aab4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" Oct 30 12:56:40.546759 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:40.546669 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b278n\" (UniqueName: \"kubernetes.io/projected/935f35f7-28c0-4d05-ab8e-1a2042d7aab4-kube-api-access-b278n\") pod \"payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj\" (UID: \"935f35f7-28c0-4d05-ab8e-1a2042d7aab4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" Oct 30 12:56:40.546971 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:40.546790 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935f35f7-28c0-4d05-ab8e-1a2042d7aab4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj\" (UID: \"935f35f7-28c0-4d05-ab8e-1a2042d7aab4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" Oct 30 12:56:40.548434 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:40.548391 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935f35f7-28c0-4d05-ab8e-1a2042d7aab4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj\" (UID: \"935f35f7-28c0-4d05-ab8e-1a2042d7aab4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" Oct 30 12:56:40.572443 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:40.572376 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b278n\" (UniqueName: \"kubernetes.io/projected/935f35f7-28c0-4d05-ab8e-1a2042d7aab4-kube-api-access-b278n\") pod \"payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj\" (UID: \"935f35f7-28c0-4d05-ab8e-1a2042d7aab4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" Oct 30 12:56:40.651288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:40.651231 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" Oct 30 12:56:40.962956 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:40.962923 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj"] Oct 30 12:56:40.963715 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:56:40.963689 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935f35f7_28c0_4d05_ab8e_1a2042d7aab4.slice/crio-fcd872b5d3ad2af50b25808881170c1508589dea47855a015935f1611feb5a1d WatchSource:0}: Error finding container fcd872b5d3ad2af50b25808881170c1508589dea47855a015935f1611feb5a1d: Status 404 returned error can't find the container with id fcd872b5d3ad2af50b25808881170c1508589dea47855a015935f1611feb5a1d Oct 30 12:56:41.279327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:41.278938 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" event={"ID":"935f35f7-28c0-4d05-ab8e-1a2042d7aab4","Type":"ContainerStarted","Data":"ece04963b3a4e083f12f1f5980b032c461b1e3470c7a472948a3c4f1165247b6"} Oct 30 12:56:41.279327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:41.279020 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" event={"ID":"935f35f7-28c0-4d05-ab8e-1a2042d7aab4","Type":"ContainerStarted","Data":"fcd872b5d3ad2af50b25808881170c1508589dea47855a015935f1611feb5a1d"} Oct 30 12:56:41.279327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:41.279168 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" Oct 30 12:56:41.321313 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:41.321261 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" podStartSLOduration=1.32123452 podStartE2EDuration="1.32123452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:41.318400461 +0000 UTC m=+3754.428243361" watchObservedRunningTime="2026-10-30 12:56:41.32123452 +0000 UTC m=+3754.431077400" Oct 30 12:56:41.484350 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:41.484321 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:42.472306 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:42.472277 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:42.908394 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:42.908361 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:43.272832 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:43.272758 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:43.419343 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:43.419306 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:43.606454 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:43.606166 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:45.103490 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:45.103455 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:45.914582 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:45.914553 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:46.026070 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:46.026031 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:46.944041 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:46.944008 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:47.458442 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:47.458413 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:47.613346 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:47.613317 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:48.953870 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:48.953835 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:49.091725 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:49.091656 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:49.223335 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:49.223231 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:49.359509 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:49.359472 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm"] Oct 30 12:56:49.369829 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:49.369764 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm" Oct 30 12:56:49.381152 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:49.381120 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm"] Oct 30 12:56:49.413455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:49.413423 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-befaa6\"" Oct 30 12:56:49.429459 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:49.429426 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-befaa6-dockercfg-tft48\"" Oct 30 12:56:49.515952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:49.515876 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm\" (UID: \"a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39\") " pod="openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm" Oct 30 12:56:49.516116 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:49.516101 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9blrn\" (UniqueName: \"kubernetes.io/projected/a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39-kube-api-access-9blrn\") pod \"payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm\" (UID: \"a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39\") " pod="openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm" Oct 30 12:56:49.617969 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:49.617934 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9blrn\" (UniqueName: \"kubernetes.io/projected/a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39-kube-api-access-9blrn\") pod \"payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm\" (UID: \"a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39\") " pod="openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm" Oct 30 12:56:49.618162 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:49.618126 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm\" (UID: \"a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39\") " pod="openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm" Oct 30 12:56:49.618864 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:49.618844 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm\" (UID: \"a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39\") " pod="openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm" Oct 30 12:56:49.641676 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:49.641642 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9blrn\" (UniqueName: \"kubernetes.io/projected/a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39-kube-api-access-9blrn\") pod \"payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm\" (UID: \"a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39\") " pod="openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm" Oct 30 12:56:49.729591 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:49.729553 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm" Oct 30 12:56:50.039871 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:50.039842 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm"] Oct 30 12:56:50.041136 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:56:50.041110 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5fd2f7f_f7b7_44cb_9993_df17cfb6cd39.slice/crio-512aa26d735ceeaea9669d238de4bb5310f70013a2be0cfa520f369523c3241b WatchSource:0}: Error finding container 512aa26d735ceeaea9669d238de4bb5310f70013a2be0cfa520f369523c3241b: Status 404 returned error can't find the container with id 512aa26d735ceeaea9669d238de4bb5310f70013a2be0cfa520f369523c3241b Oct 30 12:56:50.356809 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:50.356714 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm" event={"ID":"a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39","Type":"ContainerStarted","Data":"f01c115fbd197de2feb22b9585dff1c418b3ea5dba7b19dfeb6b10f0f9bab626"} Oct 30 12:56:50.356809 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:50.356768 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm" Oct 30 12:56:50.356809 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:50.356791 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm" event={"ID":"a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39","Type":"ContainerStarted","Data":"512aa26d735ceeaea9669d238de4bb5310f70013a2be0cfa520f369523c3241b"} Oct 30 12:56:50.401373 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:50.401323 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm" podStartSLOduration=1.4013107169999999 podStartE2EDuration="1.401310717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:50.399375002 +0000 UTC m=+3763.509217884" watchObservedRunningTime="2026-10-30 12:56:50.401310717 +0000 UTC m=+3763.511153597" Oct 30 12:56:51.308355 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:51.308312 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:52.290339 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:52.290311 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" Oct 30 12:56:53.678372 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:53.678343 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:55.568784 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:55.568750 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:57.236766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:57.236737 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:56:58.865602 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:56:58.865572 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:00.551960 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:00.551901 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:01.371829 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:01.371799 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm" Oct 30 12:57:01.838728 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:01.838692 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:02.262312 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:02.262227 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:03.885450 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:03.885420 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:05.546589 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:05.546560 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:05.814440 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:57:05.814344 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:57:07.167026 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:07.166990 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:08.966203 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:08.966159 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:10.630904 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:10.630869 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:12.464055 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:12.464020 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:14.287990 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:14.287961 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:16.161612 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:16.161576 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:17.719305 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:17.719241 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:18.400457 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:18.400426 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:19.355494 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:19.355462 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:20.984177 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:20.984138 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:22.931911 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:22.931880 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:24.717390 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:24.717353 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:25.033883 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:25.033777 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:25.511113 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:25.511069 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:26.012515 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:26.012480 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:26.158047 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:26.158016 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:26.283694 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:26.283607 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:26.816449 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:26.816417 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:28.297927 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:28.297886 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:28.468365 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:28.468332 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:30.247923 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:30.247887 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:30.791737 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:30.791704 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:30.930672 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:30.930635 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:32.149377 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:32.149344 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:32.842954 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:32.842882 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:32.993571 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:32.993539 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:33.072483 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:33.072452 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc"] Oct 30 12:57:33.077689 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:33.077665 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc" Oct 30 12:57:33.099329 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:33.099243 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc"] Oct 30 12:57:33.106285 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:33.106241 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9c3f12-dockercfg-49mwn\"" Oct 30 12:57:33.106472 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:33.106302 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9c3f12\"" Oct 30 12:57:33.208292 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:33.208236 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac48ad61-b11f-46c7-902e-87fa0d893933-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc\" (UID: \"ac48ad61-b11f-46c7-902e-87fa0d893933\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc" Oct 30 12:57:33.208792 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:33.208518 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsb9m\" (UniqueName: \"kubernetes.io/projected/ac48ad61-b11f-46c7-902e-87fa0d893933-kube-api-access-xsb9m\") pod \"payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc\" (UID: \"ac48ad61-b11f-46c7-902e-87fa0d893933\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc" Oct 30 12:57:33.309879 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:33.309844 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac48ad61-b11f-46c7-902e-87fa0d893933-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc\" (UID: \"ac48ad61-b11f-46c7-902e-87fa0d893933\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc" Oct 30 12:57:33.310072 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:33.309989 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsb9m\" (UniqueName: \"kubernetes.io/projected/ac48ad61-b11f-46c7-902e-87fa0d893933-kube-api-access-xsb9m\") pod \"payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc\" (UID: \"ac48ad61-b11f-46c7-902e-87fa0d893933\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc" Oct 30 12:57:33.310608 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:33.310590 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac48ad61-b11f-46c7-902e-87fa0d893933-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc\" (UID: \"ac48ad61-b11f-46c7-902e-87fa0d893933\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc" Oct 30 12:57:33.340550 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:33.340520 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsb9m\" (UniqueName: \"kubernetes.io/projected/ac48ad61-b11f-46c7-902e-87fa0d893933-kube-api-access-xsb9m\") pod \"payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc\" (UID: \"ac48ad61-b11f-46c7-902e-87fa0d893933\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc" Oct 30 12:57:33.393380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:33.393346 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc" Oct 30 12:57:33.715345 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:33.715316 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc"] Oct 30 12:57:33.717772 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:57:33.717740 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac48ad61_b11f_46c7_902e_87fa0d893933.slice/crio-5589dd93ded9222cfd4f598a216ae7e6fed6bcef1b4044692bf2b99309839517 WatchSource:0}: Error finding container 5589dd93ded9222cfd4f598a216ae7e6fed6bcef1b4044692bf2b99309839517: Status 404 returned error can't find the container with id 5589dd93ded9222cfd4f598a216ae7e6fed6bcef1b4044692bf2b99309839517 Oct 30 12:57:34.054347 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:34.054241 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:34.180743 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:34.180713 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:34.730818 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:34.730780 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc" event={"ID":"ac48ad61-b11f-46c7-902e-87fa0d893933","Type":"ContainerStarted","Data":"6c9a85cc056d85e6a87aa81bee98afd5cfd493b57d9e9daccb35ec4900514f23"} Oct 30 12:57:34.731380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:34.730845 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc" event={"ID":"ac48ad61-b11f-46c7-902e-87fa0d893933","Type":"ContainerStarted","Data":"5589dd93ded9222cfd4f598a216ae7e6fed6bcef1b4044692bf2b99309839517"} Oct 30 12:57:34.731380 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:34.730987 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc" Oct 30 12:57:35.817332 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:57:35.817291 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:57:35.902007 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:35.901977 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:37.563713 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:37.563682 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:39.249839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:39.249801 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:40.938800 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:40.938768 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:42.741201 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:42.741173 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:44.183943 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.183896 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc" podStartSLOduration=11.183882337 podStartE2EDuration="11.183882337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:34.810473321 +0000 UTC m=+3807.920316202" watchObservedRunningTime="2026-10-30 12:57:44.183882337 +0000 UTC m=+3817.293725218" Oct 30 12:57:44.186866 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.186844 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z"] Oct 30 12:57:44.197455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.197425 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" Oct 30 12:57:44.216573 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.216545 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z"] Oct 30 12:57:44.221852 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.221795 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-befaa6-serving-cert\"" Oct 30 12:57:44.273927 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.273896 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:44.274634 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.274605 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee65528d-b306-4f62-97fe-6314d1d9660e-maas-api-tls\") pod \"maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z\" (UID: \"ee65528d-b306-4f62-97fe-6314d1d9660e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" Oct 30 12:57:44.274839 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.274819 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf4ch\" (UniqueName: \"kubernetes.io/projected/ee65528d-b306-4f62-97fe-6314d1d9660e-kube-api-access-zf4ch\") pod \"maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z\" (UID: \"ee65528d-b306-4f62-97fe-6314d1d9660e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" Oct 30 12:57:44.274905 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.274879 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee65528d-b306-4f62-97fe-6314d1d9660e-metrics-certs\") pod \"maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z\" (UID: \"ee65528d-b306-4f62-97fe-6314d1d9660e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" Oct 30 12:57:44.275051 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.275032 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee65528d-b306-4f62-97fe-6314d1d9660e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z\" (UID: \"ee65528d-b306-4f62-97fe-6314d1d9660e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" Oct 30 12:57:44.376213 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.376182 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf4ch\" (UniqueName: \"kubernetes.io/projected/ee65528d-b306-4f62-97fe-6314d1d9660e-kube-api-access-zf4ch\") pod \"maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z\" (UID: \"ee65528d-b306-4f62-97fe-6314d1d9660e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" Oct 30 12:57:44.376403 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.376240 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee65528d-b306-4f62-97fe-6314d1d9660e-metrics-certs\") pod \"maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z\" (UID: \"ee65528d-b306-4f62-97fe-6314d1d9660e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" Oct 30 12:57:44.376403 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.376344 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee65528d-b306-4f62-97fe-6314d1d9660e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z\" (UID: \"ee65528d-b306-4f62-97fe-6314d1d9660e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" Oct 30 12:57:44.376498 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.376402 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee65528d-b306-4f62-97fe-6314d1d9660e-maas-api-tls\") pod \"maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z\" (UID: \"ee65528d-b306-4f62-97fe-6314d1d9660e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" Oct 30 12:57:44.377056 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.377034 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee65528d-b306-4f62-97fe-6314d1d9660e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z\" (UID: \"ee65528d-b306-4f62-97fe-6314d1d9660e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" Oct 30 12:57:44.378873 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.378850 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee65528d-b306-4f62-97fe-6314d1d9660e-metrics-certs\") pod \"maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z\" (UID: \"ee65528d-b306-4f62-97fe-6314d1d9660e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" Oct 30 12:57:44.378983 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.378916 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee65528d-b306-4f62-97fe-6314d1d9660e-maas-api-tls\") pod \"maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z\" (UID: \"ee65528d-b306-4f62-97fe-6314d1d9660e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" Oct 30 12:57:44.386368 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.386342 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:44.399492 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.399464 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf4ch\" (UniqueName: \"kubernetes.io/projected/ee65528d-b306-4f62-97fe-6314d1d9660e-kube-api-access-zf4ch\") pod \"maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z\" (UID: \"ee65528d-b306-4f62-97fe-6314d1d9660e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" Oct 30 12:57:44.510647 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.510556 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:44.523899 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.523871 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" Oct 30 12:57:44.836227 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:44.836192 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z"] Oct 30 12:57:44.836912 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:57:44.836884 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee65528d_b306_4f62_97fe_6314d1d9660e.slice/crio-6cd53a6e578acc801f38f6ff11df1ef0ca9d19b7b7159fe9fce266c4141b65e0 WatchSource:0}: Error finding container 6cd53a6e578acc801f38f6ff11df1ef0ca9d19b7b7159fe9fce266c4141b65e0: Status 404 returned error can't find the container with id 6cd53a6e578acc801f38f6ff11df1ef0ca9d19b7b7159fe9fce266c4141b65e0 Oct 30 12:57:45.742669 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:45.742638 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc" Oct 30 12:57:45.823916 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:45.823871 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" event={"ID":"ee65528d-b306-4f62-97fe-6314d1d9660e","Type":"ContainerStarted","Data":"28e14e773107a486b04084d5076f540f95110fe5b6f49531ac8bca228265f901"} Oct 30 12:57:45.824112 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:45.823945 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" event={"ID":"ee65528d-b306-4f62-97fe-6314d1d9660e","Type":"ContainerStarted","Data":"6cd53a6e578acc801f38f6ff11df1ef0ca9d19b7b7159fe9fce266c4141b65e0"} Oct 30 12:57:45.824112 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:45.823977 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" Oct 30 12:57:45.957568 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:45.957534 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:47.633810 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:47.633777 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:49.271210 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:49.271121 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:50.960308 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:50.960243 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:51.681196 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:51.681147 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:51.836767 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:51.836739 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" Oct 30 12:57:51.888272 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:51.888155 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" podStartSLOduration=7.535792436 podStartE2EDuration="7.888134677s" totalImagesPullingTime="352.342241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:45.867156744 +0000 UTC m=+3818.976999626" watchObservedRunningTime="2026-10-30 12:57:51.888134677 +0000 UTC m=+3824.997977558" Oct 30 12:57:52.564315 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:52.564278 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:54.233697 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:54.233660 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:55.905368 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:55.905337 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:57.492492 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:57.492453 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:57:59.034543 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:57:59.034505 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:00.760798 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:00.760764 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:02.047785 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:02.047753 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:02.229512 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:02.229478 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:02.515403 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:02.515370 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:04.269228 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:04.269192 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:05.815788 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:58:05.815741 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:58:06.040734 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:06.040697 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:07.674665 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:07.674624 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:09.477110 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:09.477078 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:11.056377 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:11.056341 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:12.663111 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:12.663077 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:14.320310 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:14.320273 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:15.971455 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:15.971425 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:17.642368 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:17.642332 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:19.409647 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:19.409602 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:21.282648 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:21.282615 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:23.020614 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:23.020563 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:24.936430 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:24.936372 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:26.827327 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:26.827287 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:27.074738 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:27.074709 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:29.281922 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:29.281886 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:31.247473 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:31.247445 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:33.167375 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:33.167344 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:34.997894 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:34.997859 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:35.815865 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:58:35.815831 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:58:36.127304 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:36.127276 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:37.367035 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:37.367001 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:37.825961 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:37.825926 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:39.365398 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:39.365367 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:41.192310 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:41.192275 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:42.883185 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:42.883157 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:43.590333 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:43.590297 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:43.757719 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:43.757683 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:44.853524 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:44.853492 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:46.676288 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:46.676240 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:48.815037 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:48.815004 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:50.501893 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:50.501861 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:50.705056 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:50.705019 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:52.564791 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:52.564761 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:54.201466 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:54.201433 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:55.772783 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:55.772745 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:57.011360 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:57.011330 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:57.481135 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:57.481103 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:58:59.070681 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:58:59.070654 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:00.866096 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:00.866060 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:02.326472 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:02.326435 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:02.451006 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:02.450974 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:02.642649 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:02.642615 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:03.531541 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:03.531496 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:04.340369 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:04.340337 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:05.815905 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:59:05.815874 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:59:06.104823 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:06.104731 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:07.606659 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:07.606622 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:07.962573 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:07.962519 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:09.594309 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:09.594272 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:11.299342 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:11.299313 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:13.317878 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:13.317844 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:15.031110 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:15.031080 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:16.783996 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:16.783967 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:18.435372 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:18.435339 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:19.086825 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.086746 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct"] Oct 30 12:59:19.091759 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.091731 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.119145 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.119115 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct"] Oct 30 12:59:19.121290 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.121264 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-befaa6\"/\"default-dockercfg-fsbnw\"" Oct 30 12:59:19.130888 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.130864 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-befaa6\"/\"test-model-befaa6-kserve-self-signed-certs\"" Oct 30 12:59:19.131061 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.131044 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-befaa6\"/\"kube-root-ca.crt\"" Oct 30 12:59:19.134537 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.134514 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-befaa6\"/\"openshift-service-ca.crt\"" Oct 30 12:59:19.149792 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.149756 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:19.233404 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.233375 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-tmp-dir\") pod \"test-model-befaa6-kserve-855d4f4c89-qhpct\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.233531 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.233497 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-model-cache\") pod \"test-model-befaa6-kserve-855d4f4c89-qhpct\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.233576 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.233541 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-home\") pod \"test-model-befaa6-kserve-855d4f4c89-qhpct\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.233576 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.233564 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-tls-certs\") pod \"test-model-befaa6-kserve-855d4f4c89-qhpct\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.233792 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.233769 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hldx7\" (UniqueName: \"kubernetes.io/projected/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-kube-api-access-hldx7\") pod \"test-model-befaa6-kserve-855d4f4c89-qhpct\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.233861 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.233824 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-dshm\") pod \"test-model-befaa6-kserve-855d4f4c89-qhpct\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.335488 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.335450 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hldx7\" (UniqueName: \"kubernetes.io/projected/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-kube-api-access-hldx7\") pod \"test-model-befaa6-kserve-855d4f4c89-qhpct\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.335488 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.335517 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-dshm\") pod \"test-model-befaa6-kserve-855d4f4c89-qhpct\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.335952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.335699 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-tmp-dir\") pod \"test-model-befaa6-kserve-855d4f4c89-qhpct\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.335952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.335781 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-model-cache\") pod \"test-model-befaa6-kserve-855d4f4c89-qhpct\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.335952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.335830 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-home\") pod \"test-model-befaa6-kserve-855d4f4c89-qhpct\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.335952 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.335859 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-tls-certs\") pod \"test-model-befaa6-kserve-855d4f4c89-qhpct\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.336926 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.336837 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-home\") pod \"test-model-befaa6-kserve-855d4f4c89-qhpct\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.337042 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.337011 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-tmp-dir\") pod \"test-model-befaa6-kserve-855d4f4c89-qhpct\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.337743 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.337720 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-model-cache\") pod \"test-model-befaa6-kserve-855d4f4c89-qhpct\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.349933 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.349450 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-tls-certs\") pod \"test-model-befaa6-kserve-855d4f4c89-qhpct\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.360771 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.360745 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-dshm\") pod \"test-model-befaa6-kserve-855d4f4c89-qhpct\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.363213 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.363090 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hldx7\" (UniqueName: \"kubernetes.io/projected/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-kube-api-access-hldx7\") pod \"test-model-befaa6-kserve-855d4f4c89-qhpct\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.434673 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.434636 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:19.793210 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:19.793179 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct"] Oct 30 12:59:19.793909 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:59:19.793884 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf26b0f59_0bb1_4015_8c3b_196a6f8f8cd2.slice/crio-ef4ce3394afc15a1965f2c2f8ecee83ef502285b2945bfdca5ec51389905c7b3 WatchSource:0}: Error finding container ef4ce3394afc15a1965f2c2f8ecee83ef502285b2945bfdca5ec51389905c7b3: Status 404 returned error can't find the container with id ef4ce3394afc15a1965f2c2f8ecee83ef502285b2945bfdca5ec51389905c7b3 Oct 30 12:59:20.257189 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:20.257158 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:20.721041 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:20.721007 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" event={"ID":"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2","Type":"ContainerStarted","Data":"5a853a069e585be2039228aee95ffbc34ac8762e12d25ca0517b32a1bb0530ff"} Oct 30 12:59:20.721241 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:20.721051 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" event={"ID":"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2","Type":"ContainerStarted","Data":"ef4ce3394afc15a1965f2c2f8ecee83ef502285b2945bfdca5ec51389905c7b3"} Oct 30 12:59:20.775790 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:20.775738 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" podStartSLOduration=1.775723654 podStartE2EDuration="1.775723654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:20.772422103 +0000 UTC m=+3913.882264985" watchObservedRunningTime="2026-10-30 12:59:20.775723654 +0000 UTC m=+3913.885566598" Oct 30 12:59:21.435291 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:21.435236 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:22.049185 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:22.049153 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:23.658832 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:23.658799 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:25.274299 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:25.274238 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:26.864695 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:26.864651 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:28.458405 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:28.458367 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:29.434832 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:29.434794 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:29.453799 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:29.453777 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:29.812700 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:29.812627 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 12:59:30.183084 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:30.183052 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:31.842793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:31.842756 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:33.544766 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:33.544716 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:33.798941 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:33.798832 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:35.269328 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:35.269273 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:35.815733 ip-10-0-138-23 kubenswrapper[4413]: E1030 12:59:35.815697 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 12:59:36.142876 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.142844 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf"] Oct 30 12:59:36.148346 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.148315 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.169912 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.169884 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf"] Oct 30 12:59:36.176528 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.176507 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1333c8\"/\"openshift-service-ca.crt\"" Oct 30 12:59:36.176671 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.176654 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1333c8\"/\"test-model-1333c8-kserve-self-signed-certs\"" Oct 30 12:59:36.176765 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.176753 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1333c8\"/\"kube-root-ca.crt\"" Oct 30 12:59:36.176845 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.176829 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1333c8\"/\"default-dockercfg-5sd9l\"" Oct 30 12:59:36.196751 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.196725 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:36.284860 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.284825 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbf6x\" (UniqueName: \"kubernetes.io/projected/a8038e2f-a57c-490f-9206-84b5c379830a-kube-api-access-mbf6x\") pod \"test-model-1333c8-kserve-5858cd6c97-x88gf\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.285299 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.284964 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-model-cache\") pod \"test-model-1333c8-kserve-5858cd6c97-x88gf\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.285299 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.285039 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8038e2f-a57c-490f-9206-84b5c379830a-tls-certs\") pod \"test-model-1333c8-kserve-5858cd6c97-x88gf\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.285299 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.285151 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-dshm\") pod \"test-model-1333c8-kserve-5858cd6c97-x88gf\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.285459 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.285362 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-tmp-dir\") pod \"test-model-1333c8-kserve-5858cd6c97-x88gf\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.285459 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.285408 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-home\") pod \"test-model-1333c8-kserve-5858cd6c97-x88gf\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.392487 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.392451 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-model-cache\") pod \"test-model-1333c8-kserve-5858cd6c97-x88gf\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.392787 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.392768 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8038e2f-a57c-490f-9206-84b5c379830a-tls-certs\") pod \"test-model-1333c8-kserve-5858cd6c97-x88gf\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.392987 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.392938 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-dshm\") pod \"test-model-1333c8-kserve-5858cd6c97-x88gf\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.393324 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.393305 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-tmp-dir\") pod \"test-model-1333c8-kserve-5858cd6c97-x88gf\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.393469 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.393454 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-home\") pod \"test-model-1333c8-kserve-5858cd6c97-x88gf\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.393747 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.393727 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbf6x\" (UniqueName: \"kubernetes.io/projected/a8038e2f-a57c-490f-9206-84b5c379830a-kube-api-access-mbf6x\") pod \"test-model-1333c8-kserve-5858cd6c97-x88gf\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.395729 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.395704 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-model-cache\") pod \"test-model-1333c8-kserve-5858cd6c97-x88gf\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.396651 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.396629 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-home\") pod \"test-model-1333c8-kserve-5858cd6c97-x88gf\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.397056 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.397022 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-tmp-dir\") pod \"test-model-1333c8-kserve-5858cd6c97-x88gf\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.411040 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.409545 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-dshm\") pod \"test-model-1333c8-kserve-5858cd6c97-x88gf\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.415499 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.413892 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8038e2f-a57c-490f-9206-84b5c379830a-tls-certs\") pod \"test-model-1333c8-kserve-5858cd6c97-x88gf\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.422010 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.421980 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbf6x\" (UniqueName: \"kubernetes.io/projected/a8038e2f-a57c-490f-9206-84b5c379830a-kube-api-access-mbf6x\") pod \"test-model-1333c8-kserve-5858cd6c97-x88gf\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.469079 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.469046 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:36.792458 ip-10-0-138-23 kubenswrapper[4413]: W1030 12:59:36.792424 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8038e2f_a57c_490f_9206_84b5c379830a.slice/crio-d03b41e0da71f98cb53074fcece6ad189ecbac8e9ea8d4864a9a562310da4f11 WatchSource:0}: Error finding container d03b41e0da71f98cb53074fcece6ad189ecbac8e9ea8d4864a9a562310da4f11: Status 404 returned error can't find the container with id d03b41e0da71f98cb53074fcece6ad189ecbac8e9ea8d4864a9a562310da4f11 Oct 30 12:59:36.793023 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.792994 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf"] Oct 30 12:59:36.858463 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.858436 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" event={"ID":"a8038e2f-a57c-490f-9206-84b5c379830a","Type":"ContainerStarted","Data":"d03b41e0da71f98cb53074fcece6ad189ecbac8e9ea8d4864a9a562310da4f11"} Oct 30 12:59:36.871876 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:36.871856 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:37.874528 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:37.874487 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" event={"ID":"a8038e2f-a57c-490f-9206-84b5c379830a","Type":"ContainerStarted","Data":"a1c0522e85627175016c3ae12dd4029239ed56af4446b05cad742cc32e30e110"} Oct 30 12:59:37.928868 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:37.928819 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" podStartSLOduration=1.928804747 podStartE2EDuration="1.928804747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:37.926545549 +0000 UTC m=+3931.036388429" watchObservedRunningTime="2026-10-30 12:59:37.928804747 +0000 UTC m=+3931.038647634" Oct 30 12:59:38.469475 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:38.469443 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:38.562582 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:38.562553 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:40.193134 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:40.193096 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:41.874096 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:41.874056 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:43.834464 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:43.834432 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:45.536158 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:45.536125 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:46.469471 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:46.469439 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:46.481881 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:46.481854 4413 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:46.959546 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:46.959520 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 12:59:47.252914 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:47.252827 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:48.940665 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:48.940631 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:50.616361 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:50.616327 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:52.246850 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:52.246819 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:54.050494 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:54.050461 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:55.741770 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:55.741737 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:57.433526 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:57.433489 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 12:59:59.064793 ip-10-0-138-23 kubenswrapper[4413]: I1030 12:59:59.064762 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:00.263022 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.262987 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98"] Oct 30 13:00:00.277477 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.277450 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" Oct 30 13:00:00.285573 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.285550 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh"] Oct 30 13:00:00.292938 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.292919 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98"] Oct 30 13:00:00.293068 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.293028 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" Oct 30 13:00:00.303696 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.303653 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld"] Oct 30 13:00:00.309875 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.309855 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" Oct 30 13:00:00.312600 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.312575 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh"] Oct 30 13:00:00.324924 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.324901 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s"] Oct 30 13:00:00.331146 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.331124 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" Oct 30 13:00:00.337217 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.337193 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld"] Oct 30 13:00:00.358217 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.358193 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx"] Oct 30 13:00:00.370632 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.370602 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" Oct 30 13:00:00.371391 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.371273 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s"] Oct 30 13:00:00.378298 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.378230 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx"] Oct 30 13:00:00.384867 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.384841 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x"] Oct 30 13:00:00.391080 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.389528 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s75hq\"" Oct 30 13:00:00.398147 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.398111 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" Oct 30 13:00:00.418879 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.418855 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x"] Oct 30 13:00:00.464615 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.464581 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83528507-6527-4816-a0b7-abdf57f86dd9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98\" (UID: \"83528507-6527-4816-a0b7-abdf57f86dd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" Oct 30 13:00:00.464785 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.464623 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c32f618-5687-4d8a-bd81-05e3881d92e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh\" (UID: \"9c32f618-5687-4d8a-bd81-05e3881d92e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" Oct 30 13:00:00.464906 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.464881 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b69pz\" (UniqueName: \"kubernetes.io/projected/83528507-6527-4816-a0b7-abdf57f86dd9-kube-api-access-b69pz\") pod \"maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98\" (UID: \"83528507-6527-4816-a0b7-abdf57f86dd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" Oct 30 13:00:00.465107 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.465088 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvc79\" (UniqueName: \"kubernetes.io/projected/8133b92d-e5ca-4a5f-94bc-9ee436f87cfc-kube-api-access-mvc79\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld\" (UID: \"8133b92d-e5ca-4a5f-94bc-9ee436f87cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" Oct 30 13:00:00.465188 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.465156 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gptmt\" (UniqueName: \"kubernetes.io/projected/52a0865a-7865-4813-a6a9-05ff39054503-kube-api-access-gptmt\") pod \"maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s\" (UID: \"52a0865a-7865-4813-a6a9-05ff39054503\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" Oct 30 13:00:00.465353 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.465332 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8133b92d-e5ca-4a5f-94bc-9ee436f87cfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld\" (UID: \"8133b92d-e5ca-4a5f-94bc-9ee436f87cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" Oct 30 13:00:00.465419 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.465378 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a0865a-7865-4813-a6a9-05ff39054503-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s\" (UID: \"52a0865a-7865-4813-a6a9-05ff39054503\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" Oct 30 13:00:00.465500 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.465478 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4bw9\" (UniqueName: \"kubernetes.io/projected/9c32f618-5687-4d8a-bd81-05e3881d92e0-kube-api-access-b4bw9\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh\" (UID: \"9c32f618-5687-4d8a-bd81-05e3881d92e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" Oct 30 13:00:00.567167 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.567074 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8133b92d-e5ca-4a5f-94bc-9ee436f87cfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld\" (UID: \"8133b92d-e5ca-4a5f-94bc-9ee436f87cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" Oct 30 13:00:00.567331 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.567169 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a0865a-7865-4813-a6a9-05ff39054503-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s\" (UID: \"52a0865a-7865-4813-a6a9-05ff39054503\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" Oct 30 13:00:00.567331 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.567278 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4bw9\" (UniqueName: \"kubernetes.io/projected/9c32f618-5687-4d8a-bd81-05e3881d92e0-kube-api-access-b4bw9\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh\" (UID: \"9c32f618-5687-4d8a-bd81-05e3881d92e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" Oct 30 13:00:00.567488 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.567470 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83528507-6527-4816-a0b7-abdf57f86dd9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98\" (UID: \"83528507-6527-4816-a0b7-abdf57f86dd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" Oct 30 13:00:00.567563 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.567512 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c32f618-5687-4d8a-bd81-05e3881d92e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh\" (UID: \"9c32f618-5687-4d8a-bd81-05e3881d92e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" Oct 30 13:00:00.567620 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.567579 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhzfc\" (UniqueName: \"kubernetes.io/projected/5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b-kube-api-access-lhzfc\") pod \"maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx\" (UID: \"5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" Oct 30 13:00:00.567678 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.567630 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b69pz\" (UniqueName: \"kubernetes.io/projected/83528507-6527-4816-a0b7-abdf57f86dd9-kube-api-access-b69pz\") pod \"maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98\" (UID: \"83528507-6527-4816-a0b7-abdf57f86dd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" Oct 30 13:00:00.567764 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.567742 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvc79\" (UniqueName: \"kubernetes.io/projected/8133b92d-e5ca-4a5f-94bc-9ee436f87cfc-kube-api-access-mvc79\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld\" (UID: \"8133b92d-e5ca-4a5f-94bc-9ee436f87cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" Oct 30 13:00:00.567855 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.567836 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx\" (UID: \"5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" Oct 30 13:00:00.567920 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.567894 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gptmt\" (UniqueName: \"kubernetes.io/projected/52a0865a-7865-4813-a6a9-05ff39054503-kube-api-access-gptmt\") pod \"maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s\" (UID: \"52a0865a-7865-4813-a6a9-05ff39054503\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" Oct 30 13:00:00.567980 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.567958 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a0865a-7865-4813-a6a9-05ff39054503-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s\" (UID: \"52a0865a-7865-4813-a6a9-05ff39054503\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" Oct 30 13:00:00.568037 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.567996 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18816891-50b2-484b-bdb7-ecb40f57097f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x\" (UID: \"18816891-50b2-484b-bdb7-ecb40f57097f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" Oct 30 13:00:00.568037 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.567999 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8133b92d-e5ca-4a5f-94bc-9ee436f87cfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld\" (UID: \"8133b92d-e5ca-4a5f-94bc-9ee436f87cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" Oct 30 13:00:00.568153 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.568070 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g786h\" (UniqueName: \"kubernetes.io/projected/18816891-50b2-484b-bdb7-ecb40f57097f-kube-api-access-g786h\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x\" (UID: \"18816891-50b2-484b-bdb7-ecb40f57097f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" Oct 30 13:00:00.568314 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.568290 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83528507-6527-4816-a0b7-abdf57f86dd9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98\" (UID: \"83528507-6527-4816-a0b7-abdf57f86dd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" Oct 30 13:00:00.568367 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.568338 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c32f618-5687-4d8a-bd81-05e3881d92e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh\" (UID: \"9c32f618-5687-4d8a-bd81-05e3881d92e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" Oct 30 13:00:00.592125 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.592089 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b69pz\" (UniqueName: \"kubernetes.io/projected/83528507-6527-4816-a0b7-abdf57f86dd9-kube-api-access-b69pz\") pod \"maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98\" (UID: \"83528507-6527-4816-a0b7-abdf57f86dd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" Oct 30 13:00:00.592420 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.592113 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvc79\" (UniqueName: \"kubernetes.io/projected/8133b92d-e5ca-4a5f-94bc-9ee436f87cfc-kube-api-access-mvc79\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld\" (UID: \"8133b92d-e5ca-4a5f-94bc-9ee436f87cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" Oct 30 13:00:00.592570 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.592235 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gptmt\" (UniqueName: \"kubernetes.io/projected/52a0865a-7865-4813-a6a9-05ff39054503-kube-api-access-gptmt\") pod \"maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s\" (UID: \"52a0865a-7865-4813-a6a9-05ff39054503\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" Oct 30 13:00:00.592570 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.592503 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4bw9\" (UniqueName: \"kubernetes.io/projected/9c32f618-5687-4d8a-bd81-05e3881d92e0-kube-api-access-b4bw9\") pod \"maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh\" (UID: \"9c32f618-5687-4d8a-bd81-05e3881d92e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" Oct 30 13:00:00.613123 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.613089 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" Oct 30 13:00:00.629334 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.629282 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" Oct 30 13:00:00.642053 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.642027 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" Oct 30 13:00:00.655212 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.655158 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" Oct 30 13:00:00.670284 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.670218 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18816891-50b2-484b-bdb7-ecb40f57097f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x\" (UID: \"18816891-50b2-484b-bdb7-ecb40f57097f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" Oct 30 13:00:00.670420 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.670299 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g786h\" (UniqueName: \"kubernetes.io/projected/18816891-50b2-484b-bdb7-ecb40f57097f-kube-api-access-g786h\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x\" (UID: \"18816891-50b2-484b-bdb7-ecb40f57097f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" Oct 30 13:00:00.670495 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.670486 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhzfc\" (UniqueName: \"kubernetes.io/projected/5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b-kube-api-access-lhzfc\") pod \"maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx\" (UID: \"5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" Oct 30 13:00:00.670557 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.670543 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx\" (UID: \"5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" Oct 30 13:00:00.671127 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.671103 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx\" (UID: \"5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" Oct 30 13:00:00.671775 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.671738 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18816891-50b2-484b-bdb7-ecb40f57097f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x\" (UID: \"18816891-50b2-484b-bdb7-ecb40f57097f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" Oct 30 13:00:00.696895 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.696122 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhzfc\" (UniqueName: \"kubernetes.io/projected/5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b-kube-api-access-lhzfc\") pod \"maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx\" (UID: \"5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" Oct 30 13:00:00.696895 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.696826 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g786h\" (UniqueName: \"kubernetes.io/projected/18816891-50b2-484b-bdb7-ecb40f57097f-kube-api-access-g786h\") pod \"maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x\" (UID: \"18816891-50b2-484b-bdb7-ecb40f57097f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" Oct 30 13:00:00.704932 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.703630 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" Oct 30 13:00:00.727279 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.726208 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" Oct 30 13:00:00.825722 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.825688 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:00.953450 ip-10-0-138-23 kubenswrapper[4413]: W1030 13:00:00.953415 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83528507_6527_4816_a0b7_abdf57f86dd9.slice/crio-c76e915b74a3bab3cc0663bcdfe9ff6b374be29444e3c9efdc9c04eccdf276ae WatchSource:0}: Error finding container c76e915b74a3bab3cc0663bcdfe9ff6b374be29444e3c9efdc9c04eccdf276ae: Status 404 returned error can't find the container with id c76e915b74a3bab3cc0663bcdfe9ff6b374be29444e3c9efdc9c04eccdf276ae Oct 30 13:00:00.955041 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:00.955017 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98"] Oct 30 13:00:01.062678 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:01.062642 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" event={"ID":"83528507-6527-4816-a0b7-abdf57f86dd9","Type":"ContainerStarted","Data":"c76e915b74a3bab3cc0663bcdfe9ff6b374be29444e3c9efdc9c04eccdf276ae"} Oct 30 13:00:01.100090 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:01.100056 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x"] Oct 30 13:00:01.106860 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:01.106833 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx"] Oct 30 13:00:01.154539 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:01.154509 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh"] Oct 30 13:00:01.193861 ip-10-0-138-23 kubenswrapper[4413]: W1030 13:00:01.193825 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18816891_50b2_484b_bdb7_ecb40f57097f.slice/crio-a06ef2903304aed94546446188cd65d2bf5edffbbc0d9c91ad87c21006766922 WatchSource:0}: Error finding container a06ef2903304aed94546446188cd65d2bf5edffbbc0d9c91ad87c21006766922: Status 404 returned error can't find the container with id a06ef2903304aed94546446188cd65d2bf5edffbbc0d9c91ad87c21006766922 Oct 30 13:00:01.194232 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:01.194207 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld"] Oct 30 13:00:01.194595 ip-10-0-138-23 kubenswrapper[4413]: W1030 13:00:01.194573 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d45cce5_ffa8_41cc_b1cb_136b43d3bc9b.slice/crio-6a4f5eb217c750b2156da345948f93cdb5367de37fc285a817417dd76e415e26 WatchSource:0}: Error finding container 6a4f5eb217c750b2156da345948f93cdb5367de37fc285a817417dd76e415e26: Status 404 returned error can't find the container with id 6a4f5eb217c750b2156da345948f93cdb5367de37fc285a817417dd76e415e26 Oct 30 13:00:01.195899 ip-10-0-138-23 kubenswrapper[4413]: W1030 13:00:01.195875 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c32f618_5687_4d8a_bd81_05e3881d92e0.slice/crio-53c145cc67ab56dfb821b8c67c4524aa53908d62adee13aaa6bb0b6aa4f1f71d WatchSource:0}: Error finding container 53c145cc67ab56dfb821b8c67c4524aa53908d62adee13aaa6bb0b6aa4f1f71d: Status 404 returned error can't find the container with id 53c145cc67ab56dfb821b8c67c4524aa53908d62adee13aaa6bb0b6aa4f1f71d Oct 30 13:00:01.197548 ip-10-0-138-23 kubenswrapper[4413]: W1030 13:00:01.197517 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8133b92d_e5ca_4a5f_94bc_9ee436f87cfc.slice/crio-1801dfa804f0bf835809be78ab26eac04e5bd8725669f3382ee8a6ac9df85c99 WatchSource:0}: Error finding container 1801dfa804f0bf835809be78ab26eac04e5bd8725669f3382ee8a6ac9df85c99: Status 404 returned error can't find the container with id 1801dfa804f0bf835809be78ab26eac04e5bd8725669f3382ee8a6ac9df85c99 Oct 30 13:00:01.236610 ip-10-0-138-23 kubenswrapper[4413]: W1030 13:00:01.236577 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a0865a_7865_4813_a6a9_05ff39054503.slice/crio-2d2a962ddb67fff252b30fc889aa84b7ee2d188c09b351394e8d052167d63471 WatchSource:0}: Error finding container 2d2a962ddb67fff252b30fc889aa84b7ee2d188c09b351394e8d052167d63471: Status 404 returned error can't find the container with id 2d2a962ddb67fff252b30fc889aa84b7ee2d188c09b351394e8d052167d63471 Oct 30 13:00:01.237571 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:01.237502 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s"] Oct 30 13:00:02.074498 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.074413 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" event={"ID":"5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b","Type":"ContainerStarted","Data":"4be61b148a6e52b8e5cdbfd5f8a468c388d66aa72b8fff3c481cf2ceba99af92"} Oct 30 13:00:02.074498 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.074475 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" event={"ID":"5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b","Type":"ContainerStarted","Data":"6a4f5eb217c750b2156da345948f93cdb5367de37fc285a817417dd76e415e26"} Oct 30 13:00:02.076339 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.076310 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" event={"ID":"83528507-6527-4816-a0b7-abdf57f86dd9","Type":"ContainerStarted","Data":"a18bfcf363dab96ec8590fee654dab68cf688a6004817428c9d54199773fc4c1"} Oct 30 13:00:02.078097 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.078070 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" event={"ID":"18816891-50b2-484b-bdb7-ecb40f57097f","Type":"ContainerStarted","Data":"6b254d6e375cb16432312d816ce6639f2cdb8133b75bc71f5b6c7689e280140c"} Oct 30 13:00:02.078219 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.078121 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" event={"ID":"18816891-50b2-484b-bdb7-ecb40f57097f","Type":"ContainerStarted","Data":"a06ef2903304aed94546446188cd65d2bf5edffbbc0d9c91ad87c21006766922"} Oct 30 13:00:02.079869 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.079850 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" event={"ID":"9c32f618-5687-4d8a-bd81-05e3881d92e0","Type":"ContainerStarted","Data":"47d6ba753cb7c074c6b565cf9a4eec3441a565e28ddc0350178a5e8a036f2a7d"} Oct 30 13:00:02.079960 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.079885 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" event={"ID":"9c32f618-5687-4d8a-bd81-05e3881d92e0","Type":"ContainerStarted","Data":"53c145cc67ab56dfb821b8c67c4524aa53908d62adee13aaa6bb0b6aa4f1f71d"} Oct 30 13:00:02.081580 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.081554 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" event={"ID":"52a0865a-7865-4813-a6a9-05ff39054503","Type":"ContainerStarted","Data":"11e3ef3fb23610284678345dfed6a0deba9dd7888fa762b3f67210f0aff6ba6f"} Oct 30 13:00:02.081715 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.081614 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" event={"ID":"52a0865a-7865-4813-a6a9-05ff39054503","Type":"ContainerStarted","Data":"2d2a962ddb67fff252b30fc889aa84b7ee2d188c09b351394e8d052167d63471"} Oct 30 13:00:02.083242 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.083221 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" event={"ID":"8133b92d-e5ca-4a5f-94bc-9ee436f87cfc","Type":"ContainerStarted","Data":"f9b8d4519c1251a33d4c9391da3e55ec185762904ee00be409d08abeb3e119eb"} Oct 30 13:00:02.083352 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.083300 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" event={"ID":"8133b92d-e5ca-4a5f-94bc-9ee436f87cfc","Type":"ContainerStarted","Data":"1801dfa804f0bf835809be78ab26eac04e5bd8725669f3382ee8a6ac9df85c99"} Oct 30 13:00:02.120482 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.120399 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" podStartSLOduration=2.120380649 podStartE2EDuration="2.120380649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:02.117076767 +0000 UTC m=+3955.226919649" watchObservedRunningTime="2026-10-30 13:00:02.120380649 +0000 UTC m=+3955.230223532" Oct 30 13:00:02.174098 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.174043 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" podStartSLOduration=2.174028967 podStartE2EDuration="2.174028967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:02.172591691 +0000 UTC m=+3955.282434571" watchObservedRunningTime="2026-10-30 13:00:02.174028967 +0000 UTC m=+3955.283871849" Oct 30 13:00:02.216557 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.216509 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" podStartSLOduration=2.216491482 podStartE2EDuration="2.216491482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:02.213593089 +0000 UTC m=+3955.323435969" watchObservedRunningTime="2026-10-30 13:00:02.216491482 +0000 UTC m=+3955.326334364" Oct 30 13:00:02.256874 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.256830 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" podStartSLOduration=2.256812699 podStartE2EDuration="2.256812699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:02.254196549 +0000 UTC m=+3955.364039430" watchObservedRunningTime="2026-10-30 13:00:02.256812699 +0000 UTC m=+3955.366655582" Oct 30 13:00:02.301761 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.301707 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" podStartSLOduration=2.301694627 podStartE2EDuration="2.301694627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:02.29675473 +0000 UTC m=+3955.406597628" watchObservedRunningTime="2026-10-30 13:00:02.301694627 +0000 UTC m=+3955.411537534" Oct 30 13:00:02.340786 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.340703 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" podStartSLOduration=2.340686437 podStartE2EDuration="2.340686437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:02.337765077 +0000 UTC m=+3955.447607969" watchObservedRunningTime="2026-10-30 13:00:02.340686437 +0000 UTC m=+3955.450529319" Oct 30 13:00:02.415037 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.415008 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:02.746361 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:02.746329 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:03.092782 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:03.092699 4413 generic.go:356] "Generic (PLEG): container finished" podID="52a0865a-7865-4813-a6a9-05ff39054503" containerID="11e3ef3fb23610284678345dfed6a0deba9dd7888fa762b3f67210f0aff6ba6f" exitCode=60 Oct 30 13:00:03.092782 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:03.092765 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" event={"ID":"52a0865a-7865-4813-a6a9-05ff39054503","Type":"ContainerDied","Data":"11e3ef3fb23610284678345dfed6a0deba9dd7888fa762b3f67210f0aff6ba6f"} Oct 30 13:00:03.093295 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:03.093136 4413 scope.go:122] "RemoveContainer" containerID="11e3ef3fb23610284678345dfed6a0deba9dd7888fa762b3f67210f0aff6ba6f" Oct 30 13:00:03.094590 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:03.094567 4413 generic.go:356] "Generic (PLEG): container finished" podID="8133b92d-e5ca-4a5f-94bc-9ee436f87cfc" containerID="f9b8d4519c1251a33d4c9391da3e55ec185762904ee00be409d08abeb3e119eb" exitCode=60 Oct 30 13:00:03.094701 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:03.094658 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" event={"ID":"8133b92d-e5ca-4a5f-94bc-9ee436f87cfc","Type":"ContainerDied","Data":"f9b8d4519c1251a33d4c9391da3e55ec185762904ee00be409d08abeb3e119eb"} Oct 30 13:00:03.094931 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:03.094914 4413 scope.go:122] "RemoveContainer" containerID="f9b8d4519c1251a33d4c9391da3e55ec185762904ee00be409d08abeb3e119eb" Oct 30 13:00:03.096781 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:03.096756 4413 generic.go:356] "Generic (PLEG): container finished" podID="5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b" containerID="4be61b148a6e52b8e5cdbfd5f8a468c388d66aa72b8fff3c481cf2ceba99af92" exitCode=60 Oct 30 13:00:03.096913 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:03.096802 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" event={"ID":"5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b","Type":"ContainerDied","Data":"4be61b148a6e52b8e5cdbfd5f8a468c388d66aa72b8fff3c481cf2ceba99af92"} Oct 30 13:00:03.097106 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:03.097092 4413 scope.go:122] "RemoveContainer" containerID="4be61b148a6e52b8e5cdbfd5f8a468c388d66aa72b8fff3c481cf2ceba99af92" Oct 30 13:00:03.099079 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:03.099060 4413 generic.go:356] "Generic (PLEG): container finished" podID="83528507-6527-4816-a0b7-abdf57f86dd9" containerID="a18bfcf363dab96ec8590fee654dab68cf688a6004817428c9d54199773fc4c1" exitCode=60 Oct 30 13:00:03.099150 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:03.099125 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" event={"ID":"83528507-6527-4816-a0b7-abdf57f86dd9","Type":"ContainerDied","Data":"a18bfcf363dab96ec8590fee654dab68cf688a6004817428c9d54199773fc4c1"} Oct 30 13:00:03.099397 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:03.099380 4413 scope.go:122] "RemoveContainer" containerID="a18bfcf363dab96ec8590fee654dab68cf688a6004817428c9d54199773fc4c1" Oct 30 13:00:03.100832 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:03.100812 4413 generic.go:356] "Generic (PLEG): container finished" podID="18816891-50b2-484b-bdb7-ecb40f57097f" containerID="6b254d6e375cb16432312d816ce6639f2cdb8133b75bc71f5b6c7689e280140c" exitCode=60 Oct 30 13:00:03.100926 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:03.100892 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" event={"ID":"18816891-50b2-484b-bdb7-ecb40f57097f","Type":"ContainerDied","Data":"6b254d6e375cb16432312d816ce6639f2cdb8133b75bc71f5b6c7689e280140c"} Oct 30 13:00:03.101055 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:03.101042 4413 scope.go:122] "RemoveContainer" containerID="6b254d6e375cb16432312d816ce6639f2cdb8133b75bc71f5b6c7689e280140c" Oct 30 13:00:03.102622 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:03.102604 4413 generic.go:356] "Generic (PLEG): container finished" podID="9c32f618-5687-4d8a-bd81-05e3881d92e0" containerID="47d6ba753cb7c074c6b565cf9a4eec3441a565e28ddc0350178a5e8a036f2a7d" exitCode=60 Oct 30 13:00:03.102714 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:03.102677 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" event={"ID":"9c32f618-5687-4d8a-bd81-05e3881d92e0","Type":"ContainerDied","Data":"47d6ba753cb7c074c6b565cf9a4eec3441a565e28ddc0350178a5e8a036f2a7d"} Oct 30 13:00:03.102846 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:03.102832 4413 scope.go:122] "RemoveContainer" containerID="47d6ba753cb7c074c6b565cf9a4eec3441a565e28ddc0350178a5e8a036f2a7d" Oct 30 13:00:04.113117 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:04.113032 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" event={"ID":"52a0865a-7865-4813-a6a9-05ff39054503","Type":"ContainerStarted","Data":"2fdc4b688ba314b0d0e782f2af5d4b4fa2eb35d207f00804430855fc041a659f"} Oct 30 13:00:04.115501 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:04.115473 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" event={"ID":"8133b92d-e5ca-4a5f-94bc-9ee436f87cfc","Type":"ContainerStarted","Data":"b858e0713abb44f8d18a1bdb766be8bf854eddd79de1c87e0dc7c3a9fb6f0f7c"} Oct 30 13:00:04.117847 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:04.117822 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" event={"ID":"5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b","Type":"ContainerStarted","Data":"a0a32dad26095d13ba3d16a6488f0fb6632fe2a67a7a85d252b8bd20366bbe9a"} Oct 30 13:00:04.120015 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:04.119994 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" event={"ID":"83528507-6527-4816-a0b7-abdf57f86dd9","Type":"ContainerStarted","Data":"367e183ab3b6c3a53eadf56550eeeb97b899128621af607a893605370383c884"} Oct 30 13:00:04.122007 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:04.121987 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" event={"ID":"18816891-50b2-484b-bdb7-ecb40f57097f","Type":"ContainerStarted","Data":"0a8d0d948f622813ea355cbbc69c56b4aa7c2472cf87803925214c909aab54de"} Oct 30 13:00:04.128928 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:04.124570 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" event={"ID":"9c32f618-5687-4d8a-bd81-05e3881d92e0","Type":"ContainerStarted","Data":"edea0d0dcf67bd791c258c0205ca9cd036de42411f79663ca6fd4aa8786af270"} Oct 30 13:00:04.367828 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:04.367747 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:04.554734 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:04.554693 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:05.139925 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.139891 4413 generic.go:356] "Generic (PLEG): container finished" podID="5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b" containerID="a0a32dad26095d13ba3d16a6488f0fb6632fe2a67a7a85d252b8bd20366bbe9a" exitCode=60 Oct 30 13:00:05.140440 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.139975 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" event={"ID":"5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b","Type":"ContainerDied","Data":"a0a32dad26095d13ba3d16a6488f0fb6632fe2a67a7a85d252b8bd20366bbe9a"} Oct 30 13:00:05.140440 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.140046 4413 scope.go:122] "RemoveContainer" containerID="4be61b148a6e52b8e5cdbfd5f8a468c388d66aa72b8fff3c481cf2ceba99af92" Oct 30 13:00:05.140515 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.140495 4413 scope.go:122] "RemoveContainer" containerID="a0a32dad26095d13ba3d16a6488f0fb6632fe2a67a7a85d252b8bd20366bbe9a" Oct 30 13:00:05.140962 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:05.140937 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx_odh-ai-gateway-infra(5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" podUID="5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b" Oct 30 13:00:05.142708 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.142678 4413 generic.go:356] "Generic (PLEG): container finished" podID="83528507-6527-4816-a0b7-abdf57f86dd9" containerID="367e183ab3b6c3a53eadf56550eeeb97b899128621af607a893605370383c884" exitCode=60 Oct 30 13:00:05.142808 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.142727 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" event={"ID":"83528507-6527-4816-a0b7-abdf57f86dd9","Type":"ContainerDied","Data":"367e183ab3b6c3a53eadf56550eeeb97b899128621af607a893605370383c884"} Oct 30 13:00:05.143096 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.143080 4413 scope.go:122] "RemoveContainer" containerID="367e183ab3b6c3a53eadf56550eeeb97b899128621af607a893605370383c884" Oct 30 13:00:05.143562 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:05.143533 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98_odh-ai-gateway-infra(83528507-6527-4816-a0b7-abdf57f86dd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" podUID="83528507-6527-4816-a0b7-abdf57f86dd9" Oct 30 13:00:05.144871 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.144849 4413 generic.go:356] "Generic (PLEG): container finished" podID="18816891-50b2-484b-bdb7-ecb40f57097f" containerID="0a8d0d948f622813ea355cbbc69c56b4aa7c2472cf87803925214c909aab54de" exitCode=60 Oct 30 13:00:05.145034 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.144937 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" event={"ID":"18816891-50b2-484b-bdb7-ecb40f57097f","Type":"ContainerDied","Data":"0a8d0d948f622813ea355cbbc69c56b4aa7c2472cf87803925214c909aab54de"} Oct 30 13:00:05.145326 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.145308 4413 scope.go:122] "RemoveContainer" containerID="0a8d0d948f622813ea355cbbc69c56b4aa7c2472cf87803925214c909aab54de" Oct 30 13:00:05.145682 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:05.145665 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x_odh-ai-gateway-infra(18816891-50b2-484b-bdb7-ecb40f57097f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" podUID="18816891-50b2-484b-bdb7-ecb40f57097f" Oct 30 13:00:05.147849 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.147826 4413 generic.go:356] "Generic (PLEG): container finished" podID="9c32f618-5687-4d8a-bd81-05e3881d92e0" containerID="edea0d0dcf67bd791c258c0205ca9cd036de42411f79663ca6fd4aa8786af270" exitCode=60 Oct 30 13:00:05.147962 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.147943 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" event={"ID":"9c32f618-5687-4d8a-bd81-05e3881d92e0","Type":"ContainerDied","Data":"edea0d0dcf67bd791c258c0205ca9cd036de42411f79663ca6fd4aa8786af270"} Oct 30 13:00:05.148289 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.148226 4413 scope.go:122] "RemoveContainer" containerID="edea0d0dcf67bd791c258c0205ca9cd036de42411f79663ca6fd4aa8786af270" Oct 30 13:00:05.148711 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:05.148685 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh_odh-ai-gateway-infra(9c32f618-5687-4d8a-bd81-05e3881d92e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" podUID="9c32f618-5687-4d8a-bd81-05e3881d92e0" Oct 30 13:00:05.150521 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.150502 4413 generic.go:356] "Generic (PLEG): container finished" podID="52a0865a-7865-4813-a6a9-05ff39054503" containerID="2fdc4b688ba314b0d0e782f2af5d4b4fa2eb35d207f00804430855fc041a659f" exitCode=60 Oct 30 13:00:05.150604 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.150558 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" event={"ID":"52a0865a-7865-4813-a6a9-05ff39054503","Type":"ContainerDied","Data":"2fdc4b688ba314b0d0e782f2af5d4b4fa2eb35d207f00804430855fc041a659f"} Oct 30 13:00:05.150860 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.150846 4413 scope.go:122] "RemoveContainer" containerID="2fdc4b688ba314b0d0e782f2af5d4b4fa2eb35d207f00804430855fc041a659f" Oct 30 13:00:05.151195 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:05.151177 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s_odh-ai-gateway-infra(52a0865a-7865-4813-a6a9-05ff39054503)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" podUID="52a0865a-7865-4813-a6a9-05ff39054503" Oct 30 13:00:05.152883 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.152858 4413 generic.go:356] "Generic (PLEG): container finished" podID="8133b92d-e5ca-4a5f-94bc-9ee436f87cfc" containerID="b858e0713abb44f8d18a1bdb766be8bf854eddd79de1c87e0dc7c3a9fb6f0f7c" exitCode=60 Oct 30 13:00:05.153004 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.152907 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" event={"ID":"8133b92d-e5ca-4a5f-94bc-9ee436f87cfc","Type":"ContainerDied","Data":"b858e0713abb44f8d18a1bdb766be8bf854eddd79de1c87e0dc7c3a9fb6f0f7c"} Oct 30 13:00:05.153413 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.153395 4413 scope.go:122] "RemoveContainer" containerID="b858e0713abb44f8d18a1bdb766be8bf854eddd79de1c87e0dc7c3a9fb6f0f7c" Oct 30 13:00:05.153703 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:05.153681 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld_odh-ai-gateway-infra(8133b92d-e5ca-4a5f-94bc-9ee436f87cfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" podUID="8133b92d-e5ca-4a5f-94bc-9ee436f87cfc" Oct 30 13:00:05.156793 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.156772 4413 scope.go:122] "RemoveContainer" containerID="a18bfcf363dab96ec8590fee654dab68cf688a6004817428c9d54199773fc4c1" Oct 30 13:00:05.172630 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.172612 4413 scope.go:122] "RemoveContainer" containerID="6b254d6e375cb16432312d816ce6639f2cdb8133b75bc71f5b6c7689e280140c" Oct 30 13:00:05.188808 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.188781 4413 scope.go:122] "RemoveContainer" containerID="47d6ba753cb7c074c6b565cf9a4eec3441a565e28ddc0350178a5e8a036f2a7d" Oct 30 13:00:05.205951 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.205929 4413 scope.go:122] "RemoveContainer" containerID="11e3ef3fb23610284678345dfed6a0deba9dd7888fa762b3f67210f0aff6ba6f" Oct 30 13:00:05.221356 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:05.221336 4413 scope.go:122] "RemoveContainer" containerID="f9b8d4519c1251a33d4c9391da3e55ec185762904ee00be409d08abeb3e119eb" Oct 30 13:00:05.820325 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:05.820283 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:00:06.164599 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:06.164559 4413 scope.go:122] "RemoveContainer" containerID="2fdc4b688ba314b0d0e782f2af5d4b4fa2eb35d207f00804430855fc041a659f" Oct 30 13:00:06.165076 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:06.165012 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s_odh-ai-gateway-infra(52a0865a-7865-4813-a6a9-05ff39054503)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" podUID="52a0865a-7865-4813-a6a9-05ff39054503" Oct 30 13:00:06.166332 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:06.166311 4413 scope.go:122] "RemoveContainer" containerID="b858e0713abb44f8d18a1bdb766be8bf854eddd79de1c87e0dc7c3a9fb6f0f7c" Oct 30 13:00:06.166714 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:06.166691 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld_odh-ai-gateway-infra(8133b92d-e5ca-4a5f-94bc-9ee436f87cfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" podUID="8133b92d-e5ca-4a5f-94bc-9ee436f87cfc" Oct 30 13:00:06.168110 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:06.168090 4413 scope.go:122] "RemoveContainer" containerID="a0a32dad26095d13ba3d16a6488f0fb6632fe2a67a7a85d252b8bd20366bbe9a" Oct 30 13:00:06.168508 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:06.168474 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx_odh-ai-gateway-infra(5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" podUID="5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b" Oct 30 13:00:06.169827 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:06.169809 4413 scope.go:122] "RemoveContainer" containerID="367e183ab3b6c3a53eadf56550eeeb97b899128621af607a893605370383c884" Oct 30 13:00:06.170225 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:06.170199 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98_odh-ai-gateway-infra(83528507-6527-4816-a0b7-abdf57f86dd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" podUID="83528507-6527-4816-a0b7-abdf57f86dd9" Oct 30 13:00:06.171766 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:06.171747 4413 scope.go:122] "RemoveContainer" containerID="0a8d0d948f622813ea355cbbc69c56b4aa7c2472cf87803925214c909aab54de" Oct 30 13:00:06.172052 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:06.172026 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x_odh-ai-gateway-infra(18816891-50b2-484b-bdb7-ecb40f57097f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" podUID="18816891-50b2-484b-bdb7-ecb40f57097f" Oct 30 13:00:06.173504 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:06.173485 4413 scope.go:122] "RemoveContainer" containerID="edea0d0dcf67bd791c258c0205ca9cd036de42411f79663ca6fd4aa8786af270" Oct 30 13:00:06.173771 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:06.173753 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh_odh-ai-gateway-infra(9c32f618-5687-4d8a-bd81-05e3881d92e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" podUID="9c32f618-5687-4d8a-bd81-05e3881d92e0" Oct 30 13:00:06.395448 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:06.395410 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:08.451894 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:08.451859 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:10.130043 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:10.130012 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:11.941427 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:11.941394 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:13.869112 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:13.869080 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:15.616366 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:15.616327 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:16.012426 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:16.012384 4413 scope.go:122] "RemoveContainer" containerID="edea0d0dcf67bd791c258c0205ca9cd036de42411f79663ca6fd4aa8786af270" Oct 30 13:00:17.018956 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:17.018918 4413 scope.go:122] "RemoveContainer" containerID="2fdc4b688ba314b0d0e782f2af5d4b4fa2eb35d207f00804430855fc041a659f" Oct 30 13:00:17.021465 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:17.021166 4413 scope.go:122] "RemoveContainer" containerID="367e183ab3b6c3a53eadf56550eeeb97b899128621af607a893605370383c884" Oct 30 13:00:17.021731 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:17.021705 4413 scope.go:122] "RemoveContainer" containerID="0a8d0d948f622813ea355cbbc69c56b4aa7c2472cf87803925214c909aab54de" Oct 30 13:00:17.022133 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:17.022111 4413 scope.go:122] "RemoveContainer" containerID="b858e0713abb44f8d18a1bdb766be8bf854eddd79de1c87e0dc7c3a9fb6f0f7c" Oct 30 13:00:17.268413 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:17.268322 4413 generic.go:356] "Generic (PLEG): container finished" podID="9c32f618-5687-4d8a-bd81-05e3881d92e0" containerID="ab8bc3831d59c580fd56c5acf5e65b67be6fb0fe6f35c00cc629faa52bf9799b" exitCode=60 Oct 30 13:00:17.268602 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:17.268408 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" event={"ID":"9c32f618-5687-4d8a-bd81-05e3881d92e0","Type":"ContainerDied","Data":"ab8bc3831d59c580fd56c5acf5e65b67be6fb0fe6f35c00cc629faa52bf9799b"} Oct 30 13:00:17.268602 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:17.268483 4413 scope.go:122] "RemoveContainer" containerID="edea0d0dcf67bd791c258c0205ca9cd036de42411f79663ca6fd4aa8786af270" Oct 30 13:00:17.268965 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:17.268949 4413 scope.go:122] "RemoveContainer" containerID="ab8bc3831d59c580fd56c5acf5e65b67be6fb0fe6f35c00cc629faa52bf9799b" Oct 30 13:00:17.269410 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:17.269380 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh_odh-ai-gateway-infra(9c32f618-5687-4d8a-bd81-05e3881d92e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" podUID="9c32f618-5687-4d8a-bd81-05e3881d92e0" Oct 30 13:00:17.546731 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:17.546637 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:18.013037 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.012650 4413 scope.go:122] "RemoveContainer" containerID="a0a32dad26095d13ba3d16a6488f0fb6632fe2a67a7a85d252b8bd20366bbe9a" Oct 30 13:00:18.279151 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.279120 4413 generic.go:356] "Generic (PLEG): container finished" podID="52a0865a-7865-4813-a6a9-05ff39054503" containerID="7c3eab26fa67aad07277b4f740b76a9a1f35d8ba1fd25034b2b784780232e000" exitCode=60 Oct 30 13:00:18.279634 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.279191 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" event={"ID":"52a0865a-7865-4813-a6a9-05ff39054503","Type":"ContainerDied","Data":"7c3eab26fa67aad07277b4f740b76a9a1f35d8ba1fd25034b2b784780232e000"} Oct 30 13:00:18.279634 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.279269 4413 scope.go:122] "RemoveContainer" containerID="2fdc4b688ba314b0d0e782f2af5d4b4fa2eb35d207f00804430855fc041a659f" Oct 30 13:00:18.279783 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.279751 4413 scope.go:122] "RemoveContainer" containerID="7c3eab26fa67aad07277b4f740b76a9a1f35d8ba1fd25034b2b784780232e000" Oct 30 13:00:18.280184 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:18.280144 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s_odh-ai-gateway-infra(52a0865a-7865-4813-a6a9-05ff39054503)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" podUID="52a0865a-7865-4813-a6a9-05ff39054503" Oct 30 13:00:18.281285 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.281263 4413 generic.go:356] "Generic (PLEG): container finished" podID="8133b92d-e5ca-4a5f-94bc-9ee436f87cfc" containerID="2361bfa70e6754637f5c66ad8d8710aaeeca32f912943ec2730c31df60ec19a5" exitCode=60 Oct 30 13:00:18.281400 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.281297 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" event={"ID":"8133b92d-e5ca-4a5f-94bc-9ee436f87cfc","Type":"ContainerDied","Data":"2361bfa70e6754637f5c66ad8d8710aaeeca32f912943ec2730c31df60ec19a5"} Oct 30 13:00:18.281762 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.281743 4413 scope.go:122] "RemoveContainer" containerID="2361bfa70e6754637f5c66ad8d8710aaeeca32f912943ec2730c31df60ec19a5" Oct 30 13:00:18.282129 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:18.282106 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld_odh-ai-gateway-infra(8133b92d-e5ca-4a5f-94bc-9ee436f87cfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" podUID="8133b92d-e5ca-4a5f-94bc-9ee436f87cfc" Oct 30 13:00:18.283425 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.283401 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" event={"ID":"83528507-6527-4816-a0b7-abdf57f86dd9","Type":"ContainerStarted","Data":"c4c40ca23c732cb2a268801f1a0c67fa9095b3fdd0aa59ec0ef7a95fd357458b"} Oct 30 13:00:18.285648 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.285624 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" event={"ID":"18816891-50b2-484b-bdb7-ecb40f57097f","Type":"ContainerStarted","Data":"156695d84b039c975dae32e5afab24a425d47638499a619bcdbd248fb0813ff9"} Oct 30 13:00:18.321598 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.321543 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh"] Oct 30 13:00:18.329016 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.328989 4413 scope.go:122] "RemoveContainer" containerID="b858e0713abb44f8d18a1bdb766be8bf854eddd79de1c87e0dc7c3a9fb6f0f7c" Oct 30 13:00:18.577154 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.577086 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" Oct 30 13:00:18.666106 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.666075 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c32f618-5687-4d8a-bd81-05e3881d92e0-kube-api-access-b4bw9\" (UniqueName: \"kubernetes.io/projected/9c32f618-5687-4d8a-bd81-05e3881d92e0-kube-api-access-b4bw9\") pod \"9c32f618-5687-4d8a-bd81-05e3881d92e0\" (UID: \"9c32f618-5687-4d8a-bd81-05e3881d92e0\") " Oct 30 13:00:18.666347 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.666158 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c32f618-5687-4d8a-bd81-05e3881d92e0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c32f618-5687-4d8a-bd81-05e3881d92e0-maas-api-service-ca\") pod \"9c32f618-5687-4d8a-bd81-05e3881d92e0\" (UID: \"9c32f618-5687-4d8a-bd81-05e3881d92e0\") " Oct 30 13:00:18.666684 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.666627 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c32f618-5687-4d8a-bd81-05e3881d92e0-maas-api-service-ca" pod "9c32f618-5687-4d8a-bd81-05e3881d92e0" (UID: "9c32f618-5687-4d8a-bd81-05e3881d92e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:18.667421 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.667396 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c32f618-5687-4d8a-bd81-05e3881d92e0-maas-api-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:00:18.670569 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.670519 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c32f618-5687-4d8a-bd81-05e3881d92e0-kube-api-access-b4bw9" pod "9c32f618-5687-4d8a-bd81-05e3881d92e0" (UID: "9c32f618-5687-4d8a-bd81-05e3881d92e0"). InnerVolumeSpecName "kube-api-access-b4bw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:18.769091 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:18.769051 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4bw9\" (UniqueName: \"kubernetes.io/projected/9c32f618-5687-4d8a-bd81-05e3881d92e0-kube-api-access-b4bw9\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:00:19.223067 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.223031 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:19.302564 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.302469 4413 generic.go:356] "Generic (PLEG): container finished" podID="5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b" containerID="5bc7911e21453edfef8a757eceac16f082a3e94d9ec9c8dedb9a38ea8959071b" exitCode=60 Oct 30 13:00:19.303016 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.302552 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" event={"ID":"5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b","Type":"ContainerDied","Data":"5bc7911e21453edfef8a757eceac16f082a3e94d9ec9c8dedb9a38ea8959071b"} Oct 30 13:00:19.303016 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.302631 4413 scope.go:122] "RemoveContainer" containerID="a0a32dad26095d13ba3d16a6488f0fb6632fe2a67a7a85d252b8bd20366bbe9a" Oct 30 13:00:19.303375 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.303353 4413 scope.go:122] "RemoveContainer" containerID="5bc7911e21453edfef8a757eceac16f082a3e94d9ec9c8dedb9a38ea8959071b" Oct 30 13:00:19.303883 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:19.303831 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx_odh-ai-gateway-infra(5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" podUID="5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b" Oct 30 13:00:19.304753 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.304730 4413 generic.go:356] "Generic (PLEG): container finished" podID="83528507-6527-4816-a0b7-abdf57f86dd9" containerID="c4c40ca23c732cb2a268801f1a0c67fa9095b3fdd0aa59ec0ef7a95fd357458b" exitCode=60 Oct 30 13:00:19.304850 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.304807 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" event={"ID":"83528507-6527-4816-a0b7-abdf57f86dd9","Type":"ContainerDied","Data":"c4c40ca23c732cb2a268801f1a0c67fa9095b3fdd0aa59ec0ef7a95fd357458b"} Oct 30 13:00:19.305337 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.305312 4413 scope.go:122] "RemoveContainer" containerID="c4c40ca23c732cb2a268801f1a0c67fa9095b3fdd0aa59ec0ef7a95fd357458b" Oct 30 13:00:19.306020 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:19.305998 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98_odh-ai-gateway-infra(83528507-6527-4816-a0b7-abdf57f86dd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" podUID="83528507-6527-4816-a0b7-abdf57f86dd9" Oct 30 13:00:19.307216 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.307186 4413 generic.go:356] "Generic (PLEG): container finished" podID="18816891-50b2-484b-bdb7-ecb40f57097f" containerID="156695d84b039c975dae32e5afab24a425d47638499a619bcdbd248fb0813ff9" exitCode=60 Oct 30 13:00:19.307344 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.307234 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" event={"ID":"18816891-50b2-484b-bdb7-ecb40f57097f","Type":"ContainerDied","Data":"156695d84b039c975dae32e5afab24a425d47638499a619bcdbd248fb0813ff9"} Oct 30 13:00:19.307724 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.307707 4413 scope.go:122] "RemoveContainer" containerID="156695d84b039c975dae32e5afab24a425d47638499a619bcdbd248fb0813ff9" Oct 30 13:00:19.308146 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:19.308114 4413 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x_odh-ai-gateway-infra(18816891-50b2-484b-bdb7-ecb40f57097f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" podUID="18816891-50b2-484b-bdb7-ecb40f57097f" Oct 30 13:00:19.308952 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.308933 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" Oct 30 13:00:19.309041 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.308950 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh" event={"ID":"9c32f618-5687-4d8a-bd81-05e3881d92e0","Type":"ContainerDied","Data":"53c145cc67ab56dfb821b8c67c4524aa53908d62adee13aaa6bb0b6aa4f1f71d"} Oct 30 13:00:19.321188 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.321162 4413 scope.go:122] "RemoveContainer" containerID="367e183ab3b6c3a53eadf56550eeeb97b899128621af607a893605370383c884" Oct 30 13:00:19.337943 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.337903 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s"] Oct 30 13:00:19.338788 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.338750 4413 scope.go:122] "RemoveContainer" containerID="0a8d0d948f622813ea355cbbc69c56b4aa7c2472cf87803925214c909aab54de" Oct 30 13:00:19.361559 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.361530 4413 scope.go:122] "RemoveContainer" containerID="ab8bc3831d59c580fd56c5acf5e65b67be6fb0fe6f35c00cc629faa52bf9799b" Oct 30 13:00:19.387463 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.387435 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld"] Oct 30 13:00:19.438336 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.436880 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x"] Oct 30 13:00:19.443323 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.442552 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh"] Oct 30 13:00:19.452308 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.451830 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-512ec5-29889420-hpwfh"] Oct 30 13:00:19.535538 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.517119 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98"] Oct 30 13:00:19.569764 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.569680 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" Oct 30 13:00:19.653785 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.653754 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" Oct 30 13:00:19.685268 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.685214 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8133b92d-e5ca-4a5f-94bc-9ee436f87cfc-kube-api-access-mvc79\" (UniqueName: \"kubernetes.io/projected/8133b92d-e5ca-4a5f-94bc-9ee436f87cfc-kube-api-access-mvc79\") pod \"8133b92d-e5ca-4a5f-94bc-9ee436f87cfc\" (UID: \"8133b92d-e5ca-4a5f-94bc-9ee436f87cfc\") " Oct 30 13:00:19.685435 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.685309 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8133b92d-e5ca-4a5f-94bc-9ee436f87cfc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8133b92d-e5ca-4a5f-94bc-9ee436f87cfc-maas-api-service-ca\") pod \"8133b92d-e5ca-4a5f-94bc-9ee436f87cfc\" (UID: \"8133b92d-e5ca-4a5f-94bc-9ee436f87cfc\") " Oct 30 13:00:19.685723 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.685701 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a0865a-7865-4813-a6a9-05ff39054503-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a0865a-7865-4813-a6a9-05ff39054503-maas-api-service-ca\") pod \"52a0865a-7865-4813-a6a9-05ff39054503\" (UID: \"52a0865a-7865-4813-a6a9-05ff39054503\") " Oct 30 13:00:19.685794 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.685772 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a0865a-7865-4813-a6a9-05ff39054503-kube-api-access-gptmt\" (UniqueName: \"kubernetes.io/projected/52a0865a-7865-4813-a6a9-05ff39054503-kube-api-access-gptmt\") pod \"52a0865a-7865-4813-a6a9-05ff39054503\" (UID: \"52a0865a-7865-4813-a6a9-05ff39054503\") " Oct 30 13:00:19.687237 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.687211 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a0865a-7865-4813-a6a9-05ff39054503-maas-api-service-ca" pod "52a0865a-7865-4813-a6a9-05ff39054503" (UID: "52a0865a-7865-4813-a6a9-05ff39054503"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:19.687237 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.687210 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8133b92d-e5ca-4a5f-94bc-9ee436f87cfc-maas-api-service-ca" pod "8133b92d-e5ca-4a5f-94bc-9ee436f87cfc" (UID: "8133b92d-e5ca-4a5f-94bc-9ee436f87cfc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:19.689214 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.689183 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a0865a-7865-4813-a6a9-05ff39054503-kube-api-access-gptmt" pod "52a0865a-7865-4813-a6a9-05ff39054503" (UID: "52a0865a-7865-4813-a6a9-05ff39054503"). InnerVolumeSpecName "kube-api-access-gptmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:19.689813 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.689791 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8133b92d-e5ca-4a5f-94bc-9ee436f87cfc-kube-api-access-mvc79" pod "8133b92d-e5ca-4a5f-94bc-9ee436f87cfc" (UID: "8133b92d-e5ca-4a5f-94bc-9ee436f87cfc"). InnerVolumeSpecName "kube-api-access-mvc79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:19.787757 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.787717 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a0865a-7865-4813-a6a9-05ff39054503-maas-api-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:00:19.787757 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.787757 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gptmt\" (UniqueName: \"kubernetes.io/projected/52a0865a-7865-4813-a6a9-05ff39054503-kube-api-access-gptmt\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:00:19.787995 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.787772 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvc79\" (UniqueName: \"kubernetes.io/projected/8133b92d-e5ca-4a5f-94bc-9ee436f87cfc-kube-api-access-mvc79\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:00:19.787995 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:19.787783 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8133b92d-e5ca-4a5f-94bc-9ee436f87cfc-maas-api-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:00:20.319437 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.319407 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" Oct 30 13:00:20.319877 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.319410 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s" event={"ID":"52a0865a-7865-4813-a6a9-05ff39054503","Type":"ContainerDied","Data":"2d2a962ddb67fff252b30fc889aa84b7ee2d188c09b351394e8d052167d63471"} Oct 30 13:00:20.319877 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.319508 4413 scope.go:122] "RemoveContainer" containerID="7c3eab26fa67aad07277b4f740b76a9a1f35d8ba1fd25034b2b784780232e000" Oct 30 13:00:20.321071 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.321041 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" event={"ID":"8133b92d-e5ca-4a5f-94bc-9ee436f87cfc","Type":"ContainerDied","Data":"1801dfa804f0bf835809be78ab26eac04e5bd8725669f3382ee8a6ac9df85c99"} Oct 30 13:00:20.321185 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.321047 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld" Oct 30 13:00:20.333005 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.332987 4413 scope.go:122] "RemoveContainer" containerID="2361bfa70e6754637f5c66ad8d8710aaeeca32f912943ec2730c31df60ec19a5" Oct 30 13:00:20.375800 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.375741 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx"] Oct 30 13:00:20.451140 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.451101 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s"] Oct 30 13:00:20.455720 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.455692 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-befaa6-29889420-lnz6s"] Oct 30 13:00:20.494214 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.494117 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld"] Oct 30 13:00:20.501326 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.501292 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f7d14-29889420-g7vld"] Oct 30 13:00:20.558049 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.558017 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" Oct 30 13:00:20.600606 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.600578 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18816891-50b2-484b-bdb7-ecb40f57097f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18816891-50b2-484b-bdb7-ecb40f57097f-maas-api-service-ca\") pod \"18816891-50b2-484b-bdb7-ecb40f57097f\" (UID: \"18816891-50b2-484b-bdb7-ecb40f57097f\") " Oct 30 13:00:20.600818 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.600801 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18816891-50b2-484b-bdb7-ecb40f57097f-kube-api-access-g786h\" (UniqueName: \"kubernetes.io/projected/18816891-50b2-484b-bdb7-ecb40f57097f-kube-api-access-g786h\") pod \"18816891-50b2-484b-bdb7-ecb40f57097f\" (UID: \"18816891-50b2-484b-bdb7-ecb40f57097f\") " Oct 30 13:00:20.600974 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.600949 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18816891-50b2-484b-bdb7-ecb40f57097f-maas-api-service-ca" pod "18816891-50b2-484b-bdb7-ecb40f57097f" (UID: "18816891-50b2-484b-bdb7-ecb40f57097f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:20.601500 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.601474 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18816891-50b2-484b-bdb7-ecb40f57097f-maas-api-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:00:20.604191 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.604159 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18816891-50b2-484b-bdb7-ecb40f57097f-kube-api-access-g786h" pod "18816891-50b2-484b-bdb7-ecb40f57097f" (UID: "18816891-50b2-484b-bdb7-ecb40f57097f"). InnerVolumeSpecName "kube-api-access-g786h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:20.614149 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.614120 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" Oct 30 13:00:20.618680 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.618658 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" Oct 30 13:00:20.702020 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.701988 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b-kube-api-access-lhzfc\" (UniqueName: \"kubernetes.io/projected/5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b-kube-api-access-lhzfc\") pod \"5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b\" (UID: \"5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b\") " Oct 30 13:00:20.702382 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.702041 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83528507-6527-4816-a0b7-abdf57f86dd9-kube-api-access-b69pz\" (UniqueName: \"kubernetes.io/projected/83528507-6527-4816-a0b7-abdf57f86dd9-kube-api-access-b69pz\") pod \"83528507-6527-4816-a0b7-abdf57f86dd9\" (UID: \"83528507-6527-4816-a0b7-abdf57f86dd9\") " Oct 30 13:00:20.702382 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.702205 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b-maas-api-service-ca\") pod \"5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b\" (UID: \"5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b\") " Oct 30 13:00:20.702382 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.702346 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83528507-6527-4816-a0b7-abdf57f86dd9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83528507-6527-4816-a0b7-abdf57f86dd9-maas-api-service-ca\") pod \"83528507-6527-4816-a0b7-abdf57f86dd9\" (UID: \"83528507-6527-4816-a0b7-abdf57f86dd9\") " Oct 30 13:00:20.703723 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.703680 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g786h\" (UniqueName: \"kubernetes.io/projected/18816891-50b2-484b-bdb7-ecb40f57097f-kube-api-access-g786h\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:00:20.704080 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.704039 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83528507-6527-4816-a0b7-abdf57f86dd9-maas-api-service-ca" pod "83528507-6527-4816-a0b7-abdf57f86dd9" (UID: "83528507-6527-4816-a0b7-abdf57f86dd9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:20.704480 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.704456 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b-maas-api-service-ca" pod "5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b" (UID: "5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:00:20.704987 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.704968 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83528507-6527-4816-a0b7-abdf57f86dd9-kube-api-access-b69pz" pod "83528507-6527-4816-a0b7-abdf57f86dd9" (UID: "83528507-6527-4816-a0b7-abdf57f86dd9"). InnerVolumeSpecName "kube-api-access-b69pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:20.705172 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.705149 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b-kube-api-access-lhzfc" pod "5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b" (UID: "5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b"). InnerVolumeSpecName "kube-api-access-lhzfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:00:20.806663 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.806528 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhzfc\" (UniqueName: \"kubernetes.io/projected/5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b-kube-api-access-lhzfc\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:00:20.806663 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.806565 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b69pz\" (UniqueName: \"kubernetes.io/projected/83528507-6527-4816-a0b7-abdf57f86dd9-kube-api-access-b69pz\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:00:20.806663 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.806581 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b-maas-api-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:00:20.806663 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:20.806595 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83528507-6527-4816-a0b7-abdf57f86dd9-maas-api-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:00:21.024744 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.024711 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a0865a-7865-4813-a6a9-05ff39054503" path="/var/lib/kubelet/pods/52a0865a-7865-4813-a6a9-05ff39054503/volumes" Oct 30 13:00:21.025352 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.025335 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8133b92d-e5ca-4a5f-94bc-9ee436f87cfc" path="/var/lib/kubelet/pods/8133b92d-e5ca-4a5f-94bc-9ee436f87cfc/volumes" Oct 30 13:00:21.025831 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.025815 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c32f618-5687-4d8a-bd81-05e3881d92e0" path="/var/lib/kubelet/pods/9c32f618-5687-4d8a-bd81-05e3881d92e0/volumes" Oct 30 13:00:21.201506 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.201475 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:21.344641 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.344610 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" Oct 30 13:00:21.344641 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.344628 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx" event={"ID":"5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b","Type":"ContainerDied","Data":"6a4f5eb217c750b2156da345948f93cdb5367de37fc285a817417dd76e415e26"} Oct 30 13:00:21.345150 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.344699 4413 scope.go:122] "RemoveContainer" containerID="5bc7911e21453edfef8a757eceac16f082a3e94d9ec9c8dedb9a38ea8959071b" Oct 30 13:00:21.347184 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.346577 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" event={"ID":"83528507-6527-4816-a0b7-abdf57f86dd9","Type":"ContainerDied","Data":"c76e915b74a3bab3cc0663bcdfe9ff6b374be29444e3c9efdc9c04eccdf276ae"} Oct 30 13:00:21.347184 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.346674 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98" Oct 30 13:00:21.349046 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.349025 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" event={"ID":"18816891-50b2-484b-bdb7-ecb40f57097f","Type":"ContainerDied","Data":"a06ef2903304aed94546446188cd65d2bf5edffbbc0d9c91ad87c21006766922"} Oct 30 13:00:21.349155 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.349092 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x" Oct 30 13:00:21.358088 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.358062 4413 scope.go:122] "RemoveContainer" containerID="c4c40ca23c732cb2a268801f1a0c67fa9095b3fdd0aa59ec0ef7a95fd357458b" Oct 30 13:00:21.369644 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.369613 4413 scope.go:122] "RemoveContainer" containerID="156695d84b039c975dae32e5afab24a425d47638499a619bcdbd248fb0813ff9" Oct 30 13:00:21.400530 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.400497 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98"] Oct 30 13:00:21.411821 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.411785 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0de4b9-29889420-hzc98"] Oct 30 13:00:21.448264 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.448210 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x"] Oct 30 13:00:21.460113 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.460022 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c1b1e8-29889420-v5w6x"] Oct 30 13:00:21.499967 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.499923 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx"] Oct 30 13:00:21.504014 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:21.503985 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-63f4e4-29889420-tjsvx"] Oct 30 13:00:23.023456 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:23.023421 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18816891-50b2-484b-bdb7-ecb40f57097f" path="/var/lib/kubelet/pods/18816891-50b2-484b-bdb7-ecb40f57097f/volumes" Oct 30 13:00:23.024103 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:23.024082 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b" path="/var/lib/kubelet/pods/5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b/volumes" Oct 30 13:00:23.024633 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:23.024611 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83528507-6527-4816-a0b7-abdf57f86dd9" path="/var/lib/kubelet/pods/83528507-6527-4816-a0b7-abdf57f86dd9/volumes" Oct 30 13:00:23.067571 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:23.067540 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:24.762686 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:24.762652 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:26.411754 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:26.411723 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:27.995034 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:27.994995 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:29.668525 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:29.668496 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:31.235015 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:31.234975 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:32.949316 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:32.949280 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:34.532690 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:34.532659 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:35.816948 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:00:35.816906 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:00:36.168755 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:36.168722 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:37.769387 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:37.769355 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:39.327897 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:39.327857 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:41.149885 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:41.149837 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:42.779501 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:42.779470 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:44.571862 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:44.571821 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:46.246316 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:46.246233 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:48.023447 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:48.023405 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:49.761165 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:49.760927 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:51.355422 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:51.355387 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:53.000235 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:53.000199 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:54.668829 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:54.668795 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:56.386176 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:56.386140 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:58.012352 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:58.012316 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:00:59.629554 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:00:59.629519 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:01.348652 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:01.348610 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:02.534593 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:02.534562 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:02.993420 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:02.993384 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:04.634017 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:04.633987 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:05.819017 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:01:05.818982 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:01:06.264026 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:06.263991 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:06.854264 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:06.854214 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:06.963644 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:06.963608 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:07.949310 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:07.949239 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:09.520159 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:09.520128 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:11.254326 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:11.254292 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:12.839613 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:12.839581 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:14.501800 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:14.501765 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:16.142009 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:16.141979 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:17.759709 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:17.759676 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:19.457754 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:19.457723 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:21.054383 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:21.054347 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:22.653485 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:22.653447 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:24.266832 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:24.266796 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:25.872498 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:25.872465 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:27.625813 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:27.625771 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:29.371327 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:29.371294 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:31.059199 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:31.059161 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:32.969774 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:32.969737 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:34.768718 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:34.768679 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:35.818565 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:01:35.818522 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:01:36.560302 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:36.560244 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:38.337463 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:38.337423 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:40.008427 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:40.008388 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:41.620010 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:41.619980 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:43.316827 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:43.316793 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:44.941510 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:44.941479 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6"] Oct 30 13:01:44.943907 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:44.941713 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" podUID="fd580504-41ed-4b72-b393-62d3cc8a49b2" containerName="istio-proxy" containerID="cri-o://e62bc2310b7c70570b60fc5548e28aae393f208d016b46028a3e8ad3924a9afb" gracePeriod=30 Oct 30 13:01:45.001129 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:45.001101 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:46.756241 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:46.756194 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:47.939300 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:47.939265 4413 patch_prober.go:29] interesting pod/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:01:47.939760 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:47.939339 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" podUID="fd580504-41ed-4b72-b393-62d3cc8a49b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:01:48.649839 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:48.649807 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:50.097046 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.097005 4413 generic.go:356] "Generic (PLEG): container finished" podID="fd580504-41ed-4b72-b393-62d3cc8a49b2" containerID="e62bc2310b7c70570b60fc5548e28aae393f208d016b46028a3e8ad3924a9afb" exitCode=0 Oct 30 13:01:50.097690 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.097079 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" event={"ID":"fd580504-41ed-4b72-b393-62d3cc8a49b2","Type":"ContainerDied","Data":"e62bc2310b7c70570b60fc5548e28aae393f208d016b46028a3e8ad3924a9afb"} Oct 30 13:01:50.229661 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.229586 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 13:01:50.308391 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.308358 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:50.313553 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.313526 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd580504-41ed-4b72-b393-62d3cc8a49b2-kube-api-access-rpxp4\" (UniqueName: \"kubernetes.io/projected/fd580504-41ed-4b72-b393-62d3cc8a49b2-kube-api-access-rpxp4\") pod \"fd580504-41ed-4b72-b393-62d3cc8a49b2\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " Oct 30 13:01:50.313719 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.313625 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-token\" (UniqueName: \"kubernetes.io/projected/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-token\") pod \"fd580504-41ed-4b72-b393-62d3cc8a49b2\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " Oct 30 13:01:50.313719 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.313676 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-podinfo\") pod \"fd580504-41ed-4b72-b393-62d3cc8a49b2\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " Oct 30 13:01:50.313840 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.313737 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-data\") pod \"fd580504-41ed-4b72-b393-62d3cc8a49b2\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " Oct 30 13:01:50.314741 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.314068 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-data" pod "fd580504-41ed-4b72-b393-62d3cc8a49b2" (UID: "fd580504-41ed-4b72-b393-62d3cc8a49b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:01:50.314741 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.314208 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd580504-41ed-4b72-b393-62d3cc8a49b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd580504-41ed-4b72-b393-62d3cc8a49b2-istiod-ca-cert\") pod \"fd580504-41ed-4b72-b393-62d3cc8a49b2\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " Oct 30 13:01:50.314741 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.314607 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd580504-41ed-4b72-b393-62d3cc8a49b2-istiod-ca-cert" pod "fd580504-41ed-4b72-b393-62d3cc8a49b2" (UID: "fd580504-41ed-4b72-b393-62d3cc8a49b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:01:50.314988 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.314860 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-credential-socket\") pod \"fd580504-41ed-4b72-b393-62d3cc8a49b2\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " Oct 30 13:01:50.315443 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.315157 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-credential-socket" pod "fd580504-41ed-4b72-b393-62d3cc8a49b2" (UID: "fd580504-41ed-4b72-b393-62d3cc8a49b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:01:50.315443 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.315285 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-workload-socket\") pod \"fd580504-41ed-4b72-b393-62d3cc8a49b2\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " Oct 30 13:01:50.315661 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.315603 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-workload-socket" pod "fd580504-41ed-4b72-b393-62d3cc8a49b2" (UID: "fd580504-41ed-4b72-b393-62d3cc8a49b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:01:50.316008 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.315977 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-envoy\") pod \"fd580504-41ed-4b72-b393-62d3cc8a49b2\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " Oct 30 13:01:50.316129 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.316111 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-workload-certs\") pod \"fd580504-41ed-4b72-b393-62d3cc8a49b2\" (UID: \"fd580504-41ed-4b72-b393-62d3cc8a49b2\") " Oct 30 13:01:50.317795 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.316595 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-workload-certs" pod "fd580504-41ed-4b72-b393-62d3cc8a49b2" (UID: "fd580504-41ed-4b72-b393-62d3cc8a49b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:01:50.317795 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.317472 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:01:50.317795 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.317495 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:01:50.317795 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.317511 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:01:50.317795 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.317527 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:01:50.317795 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.317541 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd580504-41ed-4b72-b393-62d3cc8a49b2-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:01:50.318377 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.318024 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-podinfo" pod "fd580504-41ed-4b72-b393-62d3cc8a49b2" (UID: "fd580504-41ed-4b72-b393-62d3cc8a49b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:01:50.318377 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.318135 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd580504-41ed-4b72-b393-62d3cc8a49b2-kube-api-access-rpxp4" pod "fd580504-41ed-4b72-b393-62d3cc8a49b2" (UID: "fd580504-41ed-4b72-b393-62d3cc8a49b2"). InnerVolumeSpecName "kube-api-access-rpxp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:01:50.318631 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.318601 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-token" pod "fd580504-41ed-4b72-b393-62d3cc8a49b2" (UID: "fd580504-41ed-4b72-b393-62d3cc8a49b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:01:50.318697 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.318631 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-envoy" pod "fd580504-41ed-4b72-b393-62d3cc8a49b2" (UID: "fd580504-41ed-4b72-b393-62d3cc8a49b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:01:50.419186 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.419154 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:01:50.419186 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.419185 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpxp4\" (UniqueName: \"kubernetes.io/projected/fd580504-41ed-4b72-b393-62d3cc8a49b2-kube-api-access-rpxp4\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:01:50.419780 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.419196 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:01:50.419780 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:50.419208 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd580504-41ed-4b72-b393-62d3cc8a49b2-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:01:51.106132 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:51.106099 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" Oct 30 13:01:51.106609 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:51.106120 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6" event={"ID":"fd580504-41ed-4b72-b393-62d3cc8a49b2","Type":"ContainerDied","Data":"94b8c2def249b4e1da9c1d675c416758509571898a7e56589985b48eeabeb5f0"} Oct 30 13:01:51.106609 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:51.106208 4413 scope.go:122] "RemoveContainer" containerID="e62bc2310b7c70570b60fc5548e28aae393f208d016b46028a3e8ad3924a9afb" Oct 30 13:01:51.171029 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:51.170997 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6"] Oct 30 13:01:51.183725 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:51.183701 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9d45fb-openshift-default-7c5c478599-hgjr6"] Oct 30 13:01:51.943363 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:51.943333 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:53.023695 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:53.023659 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd580504-41ed-4b72-b393-62d3cc8a49b2" path="/var/lib/kubelet/pods/fd580504-41ed-4b72-b393-62d3cc8a49b2/volumes" Oct 30 13:01:53.570395 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:53.570365 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:54.409534 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:54.409497 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:54.539376 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:54.539342 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:55.064997 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:55.064964 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:55.363244 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:55.363169 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:55.495533 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:55.495499 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:57.277820 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:57.277783 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:59.038445 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:59.038415 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:01:59.760483 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:59.760446 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj"] Oct 30 13:01:59.760690 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:01:59.760668 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" podUID="26a8eae2-4b05-4be5-9231-548a3b651406" containerName="istio-proxy" containerID="cri-o://3e5579df864ef5fddf99fe1cb3d41028e4614c558d73135fa28c135321fb88bf" gracePeriod=30 Oct 30 13:02:00.995573 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:00.995529 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:01.801675 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:01.801626 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:01.928926 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:01.928891 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:02.874602 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:02.874534 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:03.074776 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:03.074739 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:04.868631 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:04.868601 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:05.089917 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.089890 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 13:02:05.209400 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.209354 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:05.221170 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.221140 4413 generic.go:356] "Generic (PLEG): container finished" podID="26a8eae2-4b05-4be5-9231-548a3b651406" containerID="3e5579df864ef5fddf99fe1cb3d41028e4614c558d73135fa28c135321fb88bf" exitCode=0 Oct 30 13:02:05.221359 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.221204 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" Oct 30 13:02:05.221359 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.221229 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" event={"ID":"26a8eae2-4b05-4be5-9231-548a3b651406","Type":"ContainerDied","Data":"3e5579df864ef5fddf99fe1cb3d41028e4614c558d73135fa28c135321fb88bf"} Oct 30 13:02:05.221359 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.221323 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj" event={"ID":"26a8eae2-4b05-4be5-9231-548a3b651406","Type":"ContainerDied","Data":"88706a5f52efbc2fa355ab39db065f469a69ac6c5fc6afb0e6699d8930560912"} Oct 30 13:02:05.221359 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.221349 4413 scope.go:122] "RemoveContainer" containerID="3e5579df864ef5fddf99fe1cb3d41028e4614c558d73135fa28c135321fb88bf" Oct 30 13:02:05.232635 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.232615 4413 scope.go:122] "RemoveContainer" containerID="3769aa7e42ac9b067d38e25b4ec21788301d4847a24d180db71ffd8250bc865a" Oct 30 13:02:05.239056 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.239030 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26a8eae2-4b05-4be5-9231-548a3b651406-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a8eae2-4b05-4be5-9231-548a3b651406-istio-podinfo\") pod \"26a8eae2-4b05-4be5-9231-548a3b651406\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " Oct 30 13:02:05.239353 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.239300 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-credential-socket\") pod \"26a8eae2-4b05-4be5-9231-548a3b651406\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " Oct 30 13:02:05.239515 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.239353 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a8eae2-4b05-4be5-9231-548a3b651406-istio-token\" (UniqueName: \"kubernetes.io/projected/26a8eae2-4b05-4be5-9231-548a3b651406-istio-token\") pod \"26a8eae2-4b05-4be5-9231-548a3b651406\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " Oct 30 13:02:05.239515 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.239433 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a8eae2-4b05-4be5-9231-548a3b651406-kube-api-access-crxq6\" (UniqueName: \"kubernetes.io/projected/26a8eae2-4b05-4be5-9231-548a3b651406-kube-api-access-crxq6\") pod \"26a8eae2-4b05-4be5-9231-548a3b651406\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " Oct 30 13:02:05.239693 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.239672 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-credential-socket" pod "26a8eae2-4b05-4be5-9231-548a3b651406" (UID: "26a8eae2-4b05-4be5-9231-548a3b651406"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:05.239960 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.239939 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-workload-socket" pod "26a8eae2-4b05-4be5-9231-548a3b651406" (UID: "26a8eae2-4b05-4be5-9231-548a3b651406"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:05.240212 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.239474 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-workload-socket\") pod \"26a8eae2-4b05-4be5-9231-548a3b651406\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " Oct 30 13:02:05.240384 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.240299 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-istio-data\") pod \"26a8eae2-4b05-4be5-9231-548a3b651406\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " Oct 30 13:02:05.240384 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.240343 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-workload-certs\") pod \"26a8eae2-4b05-4be5-9231-548a3b651406\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " Oct 30 13:02:05.240548 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.240426 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26a8eae2-4b05-4be5-9231-548a3b651406-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a8eae2-4b05-4be5-9231-548a3b651406-istiod-ca-cert\") pod \"26a8eae2-4b05-4be5-9231-548a3b651406\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " Oct 30 13:02:05.240673 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.240653 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-istio-envoy\") pod \"26a8eae2-4b05-4be5-9231-548a3b651406\" (UID: \"26a8eae2-4b05-4be5-9231-548a3b651406\") " Oct 30 13:02:05.240772 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.240747 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-istio-data" pod "26a8eae2-4b05-4be5-9231-548a3b651406" (UID: "26a8eae2-4b05-4be5-9231-548a3b651406"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:05.241604 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.240996 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-workload-certs" pod "26a8eae2-4b05-4be5-9231-548a3b651406" (UID: "26a8eae2-4b05-4be5-9231-548a3b651406"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:05.241604 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.241366 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a8eae2-4b05-4be5-9231-548a3b651406-istiod-ca-cert" pod "26a8eae2-4b05-4be5-9231-548a3b651406" (UID: "26a8eae2-4b05-4be5-9231-548a3b651406"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:05.241604 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.241573 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:02:05.241903 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.241837 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:02:05.241903 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.241858 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:02:05.241903 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.241871 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a8eae2-4b05-4be5-9231-548a3b651406-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:02:05.241903 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.241884 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:02:05.242170 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.242142 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26a8eae2-4b05-4be5-9231-548a3b651406-istio-podinfo" pod "26a8eae2-4b05-4be5-9231-548a3b651406" (UID: "26a8eae2-4b05-4be5-9231-548a3b651406"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:02:05.242477 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.242444 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a8eae2-4b05-4be5-9231-548a3b651406-istio-token" pod "26a8eae2-4b05-4be5-9231-548a3b651406" (UID: "26a8eae2-4b05-4be5-9231-548a3b651406"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:05.242554 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.242480 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a8eae2-4b05-4be5-9231-548a3b651406-kube-api-access-crxq6" pod "26a8eae2-4b05-4be5-9231-548a3b651406" (UID: "26a8eae2-4b05-4be5-9231-548a3b651406"). InnerVolumeSpecName "kube-api-access-crxq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:05.243591 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.243567 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-istio-envoy" pod "26a8eae2-4b05-4be5-9231-548a3b651406" (UID: "26a8eae2-4b05-4be5-9231-548a3b651406"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:02:05.245647 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.245604 4413 scope.go:122] "RemoveContainer" containerID="3e5579df864ef5fddf99fe1cb3d41028e4614c558d73135fa28c135321fb88bf" Oct 30 13:02:05.246325 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:02:05.246240 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e5579df864ef5fddf99fe1cb3d41028e4614c558d73135fa28c135321fb88bf\": container with ID starting with 3e5579df864ef5fddf99fe1cb3d41028e4614c558d73135fa28c135321fb88bf not found: ID does not exist" containerID="3e5579df864ef5fddf99fe1cb3d41028e4614c558d73135fa28c135321fb88bf" Oct 30 13:02:05.246584 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.246348 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e5579df864ef5fddf99fe1cb3d41028e4614c558d73135fa28c135321fb88bf"} err="failed to get container status \"3e5579df864ef5fddf99fe1cb3d41028e4614c558d73135fa28c135321fb88bf\": rpc error: code = NotFound desc = could not find container \"3e5579df864ef5fddf99fe1cb3d41028e4614c558d73135fa28c135321fb88bf\": container with ID starting with 3e5579df864ef5fddf99fe1cb3d41028e4614c558d73135fa28c135321fb88bf not found: ID does not exist" Oct 30 13:02:05.246584 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.246381 4413 scope.go:122] "RemoveContainer" containerID="3769aa7e42ac9b067d38e25b4ec21788301d4847a24d180db71ffd8250bc865a" Oct 30 13:02:05.246831 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:02:05.246693 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3769aa7e42ac9b067d38e25b4ec21788301d4847a24d180db71ffd8250bc865a\": container with ID starting with 3769aa7e42ac9b067d38e25b4ec21788301d4847a24d180db71ffd8250bc865a not found: ID does not exist" containerID="3769aa7e42ac9b067d38e25b4ec21788301d4847a24d180db71ffd8250bc865a" Oct 30 13:02:05.246831 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.246722 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3769aa7e42ac9b067d38e25b4ec21788301d4847a24d180db71ffd8250bc865a"} err="failed to get container status \"3769aa7e42ac9b067d38e25b4ec21788301d4847a24d180db71ffd8250bc865a\": rpc error: code = NotFound desc = could not find container \"3769aa7e42ac9b067d38e25b4ec21788301d4847a24d180db71ffd8250bc865a\": container with ID starting with 3769aa7e42ac9b067d38e25b4ec21788301d4847a24d180db71ffd8250bc865a not found: ID does not exist" Oct 30 13:02:05.343760 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.343680 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a8eae2-4b05-4be5-9231-548a3b651406-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:02:05.343760 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.343710 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a8eae2-4b05-4be5-9231-548a3b651406-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:02:05.343760 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.343720 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a8eae2-4b05-4be5-9231-548a3b651406-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:02:05.343760 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.343728 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crxq6\" (UniqueName: \"kubernetes.io/projected/26a8eae2-4b05-4be5-9231-548a3b651406-kube-api-access-crxq6\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:02:05.583937 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.583905 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj"] Oct 30 13:02:05.595483 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:05.595416 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-dbe123-openshift-default-d74b4dcdf-vqtxj"] Oct 30 13:02:05.820459 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:02:05.820427 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:02:06.595481 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:06.595444 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:07.024912 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:07.024874 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26a8eae2-4b05-4be5-9231-548a3b651406" path="/var/lib/kubelet/pods/26a8eae2-4b05-4be5-9231-548a3b651406/volumes" Oct 30 13:02:08.153662 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:08.153630 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:09.919449 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:09.919406 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:11.565207 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:11.565164 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:12.668800 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:12.668763 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:12.787612 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:12.787582 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:13.302157 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:13.302125 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:14.849890 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:14.849847 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:16.728385 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:16.728349 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:18.420904 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:18.420871 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:20.625343 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:20.625240 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:22.073996 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:22.073962 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:22.802133 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:22.802097 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:24.739711 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:24.739665 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:26.739937 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:26.739904 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:28.476155 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:28.476118 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:30.458535 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:30.458495 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:32.134386 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:32.134352 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:33.125669 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:33.125637 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf"] Oct 30 13:02:33.160575 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:33.160543 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x"] Oct 30 13:02:33.290771 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:33.290740 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:33.424567 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:33.424449 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:33.929731 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:33.929695 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:35.461783 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:35.461753 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:35.842997 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:02:35.842888 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:02:36.030623 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:36.030585 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:36.232883 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:36.232851 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:36.373312 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:36.373280 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:36.815214 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:36.815166 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:37.671686 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:37.671656 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:38.126647 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.126610 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf" podUID="50eaacfb-1b6b-4cde-854b-95fd32c6dce2" containerName="payload-processing" containerID="cri-o://78722c88d588b80270c8792473c0fe94d39e3ff0a86ca4975d2a565741615e0a" gracePeriod=25 Oct 30 13:02:38.161125 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.161088 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" podUID="4fb8871c-9f64-44a2-bc8e-c110b777d677" containerName="payload-pre-processing" containerID="cri-o://d1b3fb515a5f7b4cbf8bb40528ce47462ba8e979e67bb55563e087241879ac8c" gracePeriod=25 Oct 30 13:02:38.312601 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.312567 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:38.412185 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.412156 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf" Oct 30 13:02:38.472386 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.472352 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50eaacfb-1b6b-4cde-854b-95fd32c6dce2-kube-api-access-wzww2\" (UniqueName: \"kubernetes.io/projected/50eaacfb-1b6b-4cde-854b-95fd32c6dce2-kube-api-access-wzww2\") pod \"50eaacfb-1b6b-4cde-854b-95fd32c6dce2\" (UID: \"50eaacfb-1b6b-4cde-854b-95fd32c6dce2\") " Oct 30 13:02:38.472583 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.472432 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50eaacfb-1b6b-4cde-854b-95fd32c6dce2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaacfb-1b6b-4cde-854b-95fd32c6dce2-plugins-config-volume\") pod \"50eaacfb-1b6b-4cde-854b-95fd32c6dce2\" (UID: \"50eaacfb-1b6b-4cde-854b-95fd32c6dce2\") " Oct 30 13:02:38.472839 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.472816 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50eaacfb-1b6b-4cde-854b-95fd32c6dce2-plugins-config-volume" pod "50eaacfb-1b6b-4cde-854b-95fd32c6dce2" (UID: "50eaacfb-1b6b-4cde-854b-95fd32c6dce2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:38.473195 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.473163 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaacfb-1b6b-4cde-854b-95fd32c6dce2-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:02:38.475382 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.475350 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50eaacfb-1b6b-4cde-854b-95fd32c6dce2-kube-api-access-wzww2" pod "50eaacfb-1b6b-4cde-854b-95fd32c6dce2" (UID: "50eaacfb-1b6b-4cde-854b-95fd32c6dce2"). InnerVolumeSpecName "kube-api-access-wzww2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:38.479100 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.479072 4413 generic.go:356] "Generic (PLEG): container finished" podID="50eaacfb-1b6b-4cde-854b-95fd32c6dce2" containerID="78722c88d588b80270c8792473c0fe94d39e3ff0a86ca4975d2a565741615e0a" exitCode=0 Oct 30 13:02:38.479208 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.479188 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf" Oct 30 13:02:38.479302 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.479215 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf" event={"ID":"50eaacfb-1b6b-4cde-854b-95fd32c6dce2","Type":"ContainerDied","Data":"78722c88d588b80270c8792473c0fe94d39e3ff0a86ca4975d2a565741615e0a"} Oct 30 13:02:38.479365 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.479310 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf" event={"ID":"50eaacfb-1b6b-4cde-854b-95fd32c6dce2","Type":"ContainerDied","Data":"09e4ffea0acacaa7a94476a63fafcb3ebbc4f8ce8fa51280416ea3ae7dfa8b25"} Oct 30 13:02:38.479365 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.479342 4413 scope.go:122] "RemoveContainer" containerID="78722c88d588b80270c8792473c0fe94d39e3ff0a86ca4975d2a565741615e0a" Oct 30 13:02:38.497368 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.497343 4413 scope.go:122] "RemoveContainer" containerID="78722c88d588b80270c8792473c0fe94d39e3ff0a86ca4975d2a565741615e0a" Oct 30 13:02:38.497713 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:02:38.497689 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78722c88d588b80270c8792473c0fe94d39e3ff0a86ca4975d2a565741615e0a\": container with ID starting with 78722c88d588b80270c8792473c0fe94d39e3ff0a86ca4975d2a565741615e0a not found: ID does not exist" containerID="78722c88d588b80270c8792473c0fe94d39e3ff0a86ca4975d2a565741615e0a" Oct 30 13:02:38.497777 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.497721 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78722c88d588b80270c8792473c0fe94d39e3ff0a86ca4975d2a565741615e0a"} err="failed to get container status \"78722c88d588b80270c8792473c0fe94d39e3ff0a86ca4975d2a565741615e0a\": rpc error: code = NotFound desc = could not find container \"78722c88d588b80270c8792473c0fe94d39e3ff0a86ca4975d2a565741615e0a\": container with ID starting with 78722c88d588b80270c8792473c0fe94d39e3ff0a86ca4975d2a565741615e0a not found: ID does not exist" Oct 30 13:02:38.533538 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.533509 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf"] Oct 30 13:02:38.537398 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.537375 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-eab799-574789c6bb-czsnf"] Oct 30 13:02:38.574720 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:38.574695 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzww2\" (UniqueName: \"kubernetes.io/projected/50eaacfb-1b6b-4cde-854b-95fd32c6dce2-kube-api-access-wzww2\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:02:39.023408 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:39.023376 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50eaacfb-1b6b-4cde-854b-95fd32c6dce2" path="/var/lib/kubelet/pods/50eaacfb-1b6b-4cde-854b-95fd32c6dce2/volumes" Oct 30 13:02:39.853731 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:39.853697 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:41.512533 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:41.512496 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:43.378418 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:43.378384 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:44.341573 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:44.341538 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:44.482728 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:44.482687 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:44.761092 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:44.761056 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:45.265745 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:45.265714 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:45.581779 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:45.581699 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" podUID="4fb8871c-9f64-44a2-bc8e-c110b777d677" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:02:47.054086 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:47.054052 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:48.851337 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:48.851304 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:50.478869 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:50.478829 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:52.171374 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:52.171338 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:53.841309 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:53.841280 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:55.557524 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:55.557493 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:55.581772 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:55.581738 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" podUID="4fb8871c-9f64-44a2-bc8e-c110b777d677" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:02:55.804681 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:55.804648 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9"] Oct 30 13:02:55.922958 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:55.922927 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:56.051703 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:56.051672 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:57.239865 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:57.239834 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:58.447894 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:58.447874 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" Oct 30 13:02:58.491966 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:58.491939 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fb8871c-9f64-44a2-bc8e-c110b777d677-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb8871c-9f64-44a2-bc8e-c110b777d677-plugins-config-volume\") pod \"4fb8871c-9f64-44a2-bc8e-c110b777d677\" (UID: \"4fb8871c-9f64-44a2-bc8e-c110b777d677\") " Oct 30 13:02:58.492107 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:58.492030 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb8871c-9f64-44a2-bc8e-c110b777d677-kube-api-access-pk7cr\" (UniqueName: \"kubernetes.io/projected/4fb8871c-9f64-44a2-bc8e-c110b777d677-kube-api-access-pk7cr\") pod \"4fb8871c-9f64-44a2-bc8e-c110b777d677\" (UID: \"4fb8871c-9f64-44a2-bc8e-c110b777d677\") " Oct 30 13:02:58.492469 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:58.492440 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fb8871c-9f64-44a2-bc8e-c110b777d677-plugins-config-volume" pod "4fb8871c-9f64-44a2-bc8e-c110b777d677" (UID: "4fb8871c-9f64-44a2-bc8e-c110b777d677"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:02:58.492887 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:58.492856 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fb8871c-9f64-44a2-bc8e-c110b777d677-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:02:58.497801 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:58.497774 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb8871c-9f64-44a2-bc8e-c110b777d677-kube-api-access-pk7cr" pod "4fb8871c-9f64-44a2-bc8e-c110b777d677" (UID: "4fb8871c-9f64-44a2-bc8e-c110b777d677"). InnerVolumeSpecName "kube-api-access-pk7cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:02:58.594132 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:58.594097 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk7cr\" (UniqueName: \"kubernetes.io/projected/4fb8871c-9f64-44a2-bc8e-c110b777d677-kube-api-access-pk7cr\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:02:58.645156 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:58.645130 4413 generic.go:356] "Generic (PLEG): container finished" podID="4fb8871c-9f64-44a2-bc8e-c110b777d677" containerID="d1b3fb515a5f7b4cbf8bb40528ce47462ba8e979e67bb55563e087241879ac8c" exitCode=0 Oct 30 13:02:58.645359 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:58.645184 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" Oct 30 13:02:58.645359 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:58.645211 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" event={"ID":"4fb8871c-9f64-44a2-bc8e-c110b777d677","Type":"ContainerDied","Data":"d1b3fb515a5f7b4cbf8bb40528ce47462ba8e979e67bb55563e087241879ac8c"} Oct 30 13:02:58.645359 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:58.645304 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x" event={"ID":"4fb8871c-9f64-44a2-bc8e-c110b777d677","Type":"ContainerDied","Data":"65bd78b367a66141049dd8fffc5a084e4f1c36ac34e6a0f889bd1bd8b46769c6"} Oct 30 13:02:58.645359 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:58.645340 4413 scope.go:122] "RemoveContainer" containerID="d1b3fb515a5f7b4cbf8bb40528ce47462ba8e979e67bb55563e087241879ac8c" Oct 30 13:02:58.657219 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:58.657202 4413 scope.go:122] "RemoveContainer" containerID="d1b3fb515a5f7b4cbf8bb40528ce47462ba8e979e67bb55563e087241879ac8c" Oct 30 13:02:58.657520 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:02:58.657502 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b3fb515a5f7b4cbf8bb40528ce47462ba8e979e67bb55563e087241879ac8c\": container with ID starting with d1b3fb515a5f7b4cbf8bb40528ce47462ba8e979e67bb55563e087241879ac8c not found: ID does not exist" containerID="d1b3fb515a5f7b4cbf8bb40528ce47462ba8e979e67bb55563e087241879ac8c" Oct 30 13:02:58.657587 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:58.657529 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b3fb515a5f7b4cbf8bb40528ce47462ba8e979e67bb55563e087241879ac8c"} err="failed to get container status \"d1b3fb515a5f7b4cbf8bb40528ce47462ba8e979e67bb55563e087241879ac8c\": rpc error: code = NotFound desc = could not find container \"d1b3fb515a5f7b4cbf8bb40528ce47462ba8e979e67bb55563e087241879ac8c\": container with ID starting with d1b3fb515a5f7b4cbf8bb40528ce47462ba8e979e67bb55563e087241879ac8c not found: ID does not exist" Oct 30 13:02:58.697799 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:58.697739 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x"] Oct 30 13:02:58.701890 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:58.701866 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-eab799-6cd45b65f-97l7x"] Oct 30 13:02:58.840631 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:58.840588 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:02:59.023960 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:02:59.023882 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fb8871c-9f64-44a2-bc8e-c110b777d677" path="/var/lib/kubelet/pods/4fb8871c-9f64-44a2-bc8e-c110b777d677/volumes" Oct 30 13:03:00.462579 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:00.462544 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:00.805981 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:00.805864 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" podUID="c59d84af-0bbf-4a53-bbe0-07e80541bb86" containerName="payload-pre-processing" containerID="cri-o://cce967bc67ef68aa5ae6ffdf3dc40be89cc667c2b67a98f842248ef7a5e87128" gracePeriod=25 Oct 30 13:03:02.209549 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:02.209511 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:02.778007 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:02.777975 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:03.936885 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:03.936850 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:04.497544 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:04.497509 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz"] Oct 30 13:03:04.497768 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:04.497745 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" podUID="b8fc0118-77f2-457b-8a58-82882ee852d0" containerName="istio-proxy" containerID="cri-o://4a17b7d141409fc63ceb2e44294c9e5ceb466859a1bc9875c364c08d466ec74f" gracePeriod=30 Oct 30 13:03:05.535099 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:05.535071 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:05.816626 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:03:05.816538 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:03:06.301387 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:06.301357 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:06.855633 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:06.855599 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7"] Oct 30 13:03:06.856141 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:06.855836 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" podUID="fd2d9a96-1d2e-4eee-946a-d26618a955ca" containerName="maas-api" containerID="cri-o://894357e69d40d4f2651a451f8dc0c12aef8971fea1fc17c8fd5ccde859ea7d77" gracePeriod=30 Oct 30 13:03:06.927121 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:06.927090 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:07.192719 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.192692 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 13:03:07.334493 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.334456 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd2d9a96-1d2e-4eee-946a-d26618a955ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2d9a96-1d2e-4eee-946a-d26618a955ca-openshift-service-ca\") pod \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\" (UID: \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\") " Oct 30 13:03:07.334696 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.334522 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd2d9a96-1d2e-4eee-946a-d26618a955ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2d9a96-1d2e-4eee-946a-d26618a955ca-metrics-certs\") pod \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\" (UID: \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\") " Oct 30 13:03:07.334696 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.334587 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd2d9a96-1d2e-4eee-946a-d26618a955ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd2d9a96-1d2e-4eee-946a-d26618a955ca-maas-api-tls\") pod \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\" (UID: \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\") " Oct 30 13:03:07.334826 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.334810 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2d9a96-1d2e-4eee-946a-d26618a955ca-kube-api-access-66bcj\" (UniqueName: \"kubernetes.io/projected/fd2d9a96-1d2e-4eee-946a-d26618a955ca-kube-api-access-66bcj\") pod \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\" (UID: \"fd2d9a96-1d2e-4eee-946a-d26618a955ca\") " Oct 30 13:03:07.334896 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.334825 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2d9a96-1d2e-4eee-946a-d26618a955ca-openshift-service-ca" pod "fd2d9a96-1d2e-4eee-946a-d26618a955ca" (UID: "fd2d9a96-1d2e-4eee-946a-d26618a955ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:03:07.335687 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.335658 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2d9a96-1d2e-4eee-946a-d26618a955ca-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:07.336944 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.336919 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd2d9a96-1d2e-4eee-946a-d26618a955ca-metrics-certs" pod "fd2d9a96-1d2e-4eee-946a-d26618a955ca" (UID: "fd2d9a96-1d2e-4eee-946a-d26618a955ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:03:07.337054 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.336946 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd2d9a96-1d2e-4eee-946a-d26618a955ca-maas-api-tls" pod "fd2d9a96-1d2e-4eee-946a-d26618a955ca" (UID: "fd2d9a96-1d2e-4eee-946a-d26618a955ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:03:07.337312 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.337289 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2d9a96-1d2e-4eee-946a-d26618a955ca-kube-api-access-66bcj" pod "fd2d9a96-1d2e-4eee-946a-d26618a955ca" (UID: "fd2d9a96-1d2e-4eee-946a-d26618a955ca"). InnerVolumeSpecName "kube-api-access-66bcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:07.438086 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.437986 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66bcj\" (UniqueName: \"kubernetes.io/projected/fd2d9a96-1d2e-4eee-946a-d26618a955ca-kube-api-access-66bcj\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:07.438086 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.438027 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd2d9a96-1d2e-4eee-946a-d26618a955ca-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:07.438086 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.438041 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd2d9a96-1d2e-4eee-946a-d26618a955ca-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:07.473615 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.473582 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:07.592547 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.592514 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:07.717713 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.717625 4413 generic.go:356] "Generic (PLEG): container finished" podID="fd2d9a96-1d2e-4eee-946a-d26618a955ca" containerID="894357e69d40d4f2651a451f8dc0c12aef8971fea1fc17c8fd5ccde859ea7d77" exitCode=0 Oct 30 13:03:07.717713 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.717678 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" Oct 30 13:03:07.717713 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.717705 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" event={"ID":"fd2d9a96-1d2e-4eee-946a-d26618a955ca","Type":"ContainerDied","Data":"894357e69d40d4f2651a451f8dc0c12aef8971fea1fc17c8fd5ccde859ea7d77"} Oct 30 13:03:07.717982 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.717764 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7" event={"ID":"fd2d9a96-1d2e-4eee-946a-d26618a955ca","Type":"ContainerDied","Data":"2a69b26e2c67d8d62163be3a56178e1739059c9f9dc87e9473da75b307b6c9dc"} Oct 30 13:03:07.717982 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.717790 4413 scope.go:122] "RemoveContainer" containerID="894357e69d40d4f2651a451f8dc0c12aef8971fea1fc17c8fd5ccde859ea7d77" Oct 30 13:03:07.731167 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.731139 4413 scope.go:122] "RemoveContainer" containerID="894357e69d40d4f2651a451f8dc0c12aef8971fea1fc17c8fd5ccde859ea7d77" Oct 30 13:03:07.731578 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:03:07.731546 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"894357e69d40d4f2651a451f8dc0c12aef8971fea1fc17c8fd5ccde859ea7d77\": container with ID starting with 894357e69d40d4f2651a451f8dc0c12aef8971fea1fc17c8fd5ccde859ea7d77 not found: ID does not exist" containerID="894357e69d40d4f2651a451f8dc0c12aef8971fea1fc17c8fd5ccde859ea7d77" Oct 30 13:03:07.731719 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.731591 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"894357e69d40d4f2651a451f8dc0c12aef8971fea1fc17c8fd5ccde859ea7d77"} err="failed to get container status \"894357e69d40d4f2651a451f8dc0c12aef8971fea1fc17c8fd5ccde859ea7d77\": rpc error: code = NotFound desc = could not find container \"894357e69d40d4f2651a451f8dc0c12aef8971fea1fc17c8fd5ccde859ea7d77\": container with ID starting with 894357e69d40d4f2651a451f8dc0c12aef8971fea1fc17c8fd5ccde859ea7d77 not found: ID does not exist" Oct 30 13:03:07.785242 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.785211 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7"] Oct 30 13:03:07.797242 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:07.797209 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-eab799-7f655c44db-kczd7"] Oct 30 13:03:08.085923 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:08.085843 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:09.021898 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.021823 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd2d9a96-1d2e-4eee-946a-d26618a955ca" path="/var/lib/kubelet/pods/fd2d9a96-1d2e-4eee-946a-d26618a955ca/volumes" Oct 30 13:03:09.344526 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.344449 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:09.639825 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.639792 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" podUID="c59d84af-0bbf-4a53-bbe0-07e80541bb86" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:03:09.748279 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.748207 4413 generic.go:356] "Generic (PLEG): container finished" podID="b8fc0118-77f2-457b-8a58-82882ee852d0" containerID="4a17b7d141409fc63ceb2e44294c9e5ceb466859a1bc9875c364c08d466ec74f" exitCode=0 Oct 30 13:03:09.748466 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.748284 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" event={"ID":"b8fc0118-77f2-457b-8a58-82882ee852d0","Type":"ContainerDied","Data":"4a17b7d141409fc63ceb2e44294c9e5ceb466859a1bc9875c364c08d466ec74f"} Oct 30 13:03:09.798873 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.798845 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.880643 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-credential-socket\") pod \"b8fc0118-77f2-457b-8a58-82882ee852d0\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.881147 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-credential-socket" pod "b8fc0118-77f2-457b-8a58-82882ee852d0" (UID: "b8fc0118-77f2-457b-8a58-82882ee852d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.881192 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-token\" (UniqueName: \"kubernetes.io/projected/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-token\") pod \"b8fc0118-77f2-457b-8a58-82882ee852d0\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.881339 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-data\") pod \"b8fc0118-77f2-457b-8a58-82882ee852d0\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.881431 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-workload-socket\") pod \"b8fc0118-77f2-457b-8a58-82882ee852d0\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.881498 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-data" pod "b8fc0118-77f2-457b-8a58-82882ee852d0" (UID: "b8fc0118-77f2-457b-8a58-82882ee852d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.881547 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-workload-certs\") pod \"b8fc0118-77f2-457b-8a58-82882ee852d0\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.881604 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8fc0118-77f2-457b-8a58-82882ee852d0-kube-api-access-8g9fn\" (UniqueName: \"kubernetes.io/projected/b8fc0118-77f2-457b-8a58-82882ee852d0-kube-api-access-8g9fn\") pod \"b8fc0118-77f2-457b-8a58-82882ee852d0\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.882091 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-workload-certs" pod "b8fc0118-77f2-457b-8a58-82882ee852d0" (UID: "b8fc0118-77f2-457b-8a58-82882ee852d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.882175 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-podinfo\") pod \"b8fc0118-77f2-457b-8a58-82882ee852d0\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.882230 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-workload-socket" pod "b8fc0118-77f2-457b-8a58-82882ee852d0" (UID: "b8fc0118-77f2-457b-8a58-82882ee852d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.882491 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-envoy\") pod \"b8fc0118-77f2-457b-8a58-82882ee852d0\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.882547 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8fc0118-77f2-457b-8a58-82882ee852d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8fc0118-77f2-457b-8a58-82882ee852d0-istiod-ca-cert\") pod \"b8fc0118-77f2-457b-8a58-82882ee852d0\" (UID: \"b8fc0118-77f2-457b-8a58-82882ee852d0\") " Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.882981 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8fc0118-77f2-457b-8a58-82882ee852d0-istiod-ca-cert" pod "b8fc0118-77f2-457b-8a58-82882ee852d0" (UID: "b8fc0118-77f2-457b-8a58-82882ee852d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.883361 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.883383 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.883399 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.883413 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8fc0118-77f2-457b-8a58-82882ee852d0-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.883426 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.884114 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-token" pod "b8fc0118-77f2-457b-8a58-82882ee852d0" (UID: "b8fc0118-77f2-457b-8a58-82882ee852d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.884630 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8fc0118-77f2-457b-8a58-82882ee852d0-kube-api-access-8g9fn" pod "b8fc0118-77f2-457b-8a58-82882ee852d0" (UID: "b8fc0118-77f2-457b-8a58-82882ee852d0"). InnerVolumeSpecName "kube-api-access-8g9fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.885517 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-envoy" pod "b8fc0118-77f2-457b-8a58-82882ee852d0" (UID: "b8fc0118-77f2-457b-8a58-82882ee852d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:03:09.886335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.885762 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-podinfo" pod "b8fc0118-77f2-457b-8a58-82882ee852d0" (UID: "b8fc0118-77f2-457b-8a58-82882ee852d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:03:09.985336 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.985305 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:09.985336 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.985328 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:09.985336 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.985338 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g9fn\" (UniqueName: \"kubernetes.io/projected/b8fc0118-77f2-457b-8a58-82882ee852d0-kube-api-access-8g9fn\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:09.985580 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:09.985348 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8fc0118-77f2-457b-8a58-82882ee852d0-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:10.757670 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:10.757634 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" event={"ID":"b8fc0118-77f2-457b-8a58-82882ee852d0","Type":"ContainerDied","Data":"d86b1c4c4f0db48133955136431b0829fe8ed7e9197828edc83aaba0a112d716"} Oct 30 13:03:10.758146 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:10.757683 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz" Oct 30 13:03:10.758146 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:10.757702 4413 scope.go:122] "RemoveContainer" containerID="4a17b7d141409fc63ceb2e44294c9e5ceb466859a1bc9875c364c08d466ec74f" Oct 30 13:03:10.823231 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:10.823194 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz"] Oct 30 13:03:10.834997 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:10.834963 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-bc9xz"] Oct 30 13:03:11.021670 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:11.021581 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8fc0118-77f2-457b-8a58-82882ee852d0" path="/var/lib/kubelet/pods/b8fc0118-77f2-457b-8a58-82882ee852d0/volumes" Oct 30 13:03:11.052948 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:11.052919 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:12.635610 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:12.635575 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:14.132520 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:14.132481 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:14.261638 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:14.261588 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:14.566194 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:14.566105 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:16.172987 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:16.172957 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:17.851744 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:17.851714 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:19.396287 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:19.396234 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:19.639486 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:19.639445 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" podUID="c59d84af-0bbf-4a53-bbe0-07e80541bb86" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:03:20.962579 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:20.962545 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:21.122804 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:21.122775 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" Oct 30 13:03:21.255938 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:21.255905 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c59d84af-0bbf-4a53-bbe0-07e80541bb86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c59d84af-0bbf-4a53-bbe0-07e80541bb86-plugins-config-volume\") pod \"c59d84af-0bbf-4a53-bbe0-07e80541bb86\" (UID: \"c59d84af-0bbf-4a53-bbe0-07e80541bb86\") " Oct 30 13:03:21.256202 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:21.256187 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c59d84af-0bbf-4a53-bbe0-07e80541bb86-kube-api-access-x5b7c\" (UniqueName: \"kubernetes.io/projected/c59d84af-0bbf-4a53-bbe0-07e80541bb86-kube-api-access-x5b7c\") pod \"c59d84af-0bbf-4a53-bbe0-07e80541bb86\" (UID: \"c59d84af-0bbf-4a53-bbe0-07e80541bb86\") " Oct 30 13:03:21.256328 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:21.256305 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59d84af-0bbf-4a53-bbe0-07e80541bb86-plugins-config-volume" pod "c59d84af-0bbf-4a53-bbe0-07e80541bb86" (UID: "c59d84af-0bbf-4a53-bbe0-07e80541bb86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:03:21.256691 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:21.256675 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c59d84af-0bbf-4a53-bbe0-07e80541bb86-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:21.258495 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:21.258470 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59d84af-0bbf-4a53-bbe0-07e80541bb86-kube-api-access-x5b7c" pod "c59d84af-0bbf-4a53-bbe0-07e80541bb86" (UID: "c59d84af-0bbf-4a53-bbe0-07e80541bb86"). InnerVolumeSpecName "kube-api-access-x5b7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:21.357880 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:21.357800 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5b7c\" (UniqueName: \"kubernetes.io/projected/c59d84af-0bbf-4a53-bbe0-07e80541bb86-kube-api-access-x5b7c\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:21.845837 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:21.845798 4413 generic.go:356] "Generic (PLEG): container finished" podID="c59d84af-0bbf-4a53-bbe0-07e80541bb86" containerID="cce967bc67ef68aa5ae6ffdf3dc40be89cc667c2b67a98f842248ef7a5e87128" exitCode=0 Oct 30 13:03:21.846082 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:21.845851 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" event={"ID":"c59d84af-0bbf-4a53-bbe0-07e80541bb86","Type":"ContainerDied","Data":"cce967bc67ef68aa5ae6ffdf3dc40be89cc667c2b67a98f842248ef7a5e87128"} Oct 30 13:03:21.846082 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:21.845867 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" Oct 30 13:03:21.846082 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:21.845932 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9" event={"ID":"c59d84af-0bbf-4a53-bbe0-07e80541bb86","Type":"ContainerDied","Data":"0dc104249ce5e5ff7d80be119e823aa7a6e210cc51a494ab81148b1f74b48c5b"} Oct 30 13:03:21.846082 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:21.845963 4413 scope.go:122] "RemoveContainer" containerID="cce967bc67ef68aa5ae6ffdf3dc40be89cc667c2b67a98f842248ef7a5e87128" Oct 30 13:03:21.863663 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:21.863638 4413 scope.go:122] "RemoveContainer" containerID="cce967bc67ef68aa5ae6ffdf3dc40be89cc667c2b67a98f842248ef7a5e87128" Oct 30 13:03:21.863953 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:03:21.863936 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce967bc67ef68aa5ae6ffdf3dc40be89cc667c2b67a98f842248ef7a5e87128\": container with ID starting with cce967bc67ef68aa5ae6ffdf3dc40be89cc667c2b67a98f842248ef7a5e87128 not found: ID does not exist" containerID="cce967bc67ef68aa5ae6ffdf3dc40be89cc667c2b67a98f842248ef7a5e87128" Oct 30 13:03:21.864001 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:21.863962 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce967bc67ef68aa5ae6ffdf3dc40be89cc667c2b67a98f842248ef7a5e87128"} err="failed to get container status \"cce967bc67ef68aa5ae6ffdf3dc40be89cc667c2b67a98f842248ef7a5e87128\": rpc error: code = NotFound desc = could not find container \"cce967bc67ef68aa5ae6ffdf3dc40be89cc667c2b67a98f842248ef7a5e87128\": container with ID starting with cce967bc67ef68aa5ae6ffdf3dc40be89cc667c2b67a98f842248ef7a5e87128 not found: ID does not exist" Oct 30 13:03:21.900300 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:21.900276 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9"] Oct 30 13:03:21.911669 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:21.911645 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ecb3aa-5cf4f4d675-5w5k9"] Oct 30 13:03:22.651029 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:22.650994 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:23.021298 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:23.021212 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c59d84af-0bbf-4a53-bbe0-07e80541bb86" path="/var/lib/kubelet/pods/c59d84af-0bbf-4a53-bbe0-07e80541bb86/volumes" Oct 30 13:03:24.287721 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:24.287692 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:25.626967 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:25.626930 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m"] Oct 30 13:03:25.659474 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:25.659443 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7"] Oct 30 13:03:25.835952 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:25.835912 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:26.069426 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:26.069328 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:26.372336 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:26.363737 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:27.946602 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:27.946540 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:30.053652 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:30.053613 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:30.627660 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:30.627580 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" podUID="fe249b4d-be8a-4135-acfc-568903715ef6" containerName="payload-processing" containerID="cri-o://e25e452e6225db4a7b91f13763d2716e412a6bcd57f3abe66605dccddcdbe2c6" gracePeriod=25 Oct 30 13:03:30.660122 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:30.660082 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" podUID="ed73650f-c307-4db1-80c1-9b129dd488c4" containerName="payload-pre-processing" containerID="cri-o://48ade5fc144d34274f82e39b54d8a78aade6d47cdd6339a946beb984ff3dbcee" gracePeriod=25 Oct 30 13:03:31.807178 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:31.807143 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:33.481489 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:33.481450 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:35.056358 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:35.056328 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:35.820078 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:03:35.820044 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:03:36.768743 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:36.768705 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:38.437473 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:38.437442 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:39.584842 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:39.584799 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" podUID="ed73650f-c307-4db1-80c1-9b129dd488c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:03:39.585352 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:39.584802 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" podUID="fe249b4d-be8a-4135-acfc-568903715ef6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:03:40.031226 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:40.031195 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:41.771427 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:41.771399 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:43.134933 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:43.134903 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479"] Oct 30 13:03:43.135496 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:43.135110 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" podUID="8c843f24-0220-4063-8d6d-d1f84196d9b2" containerName="maas-api" containerID="cri-o://a43fe32318610c421d56f6785362614143c4bce112bb7e64cc52422e85efecf4" gracePeriod=30 Oct 30 13:03:43.225352 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:43.225321 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:43.428799 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:43.428770 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" Oct 30 13:03:43.563594 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:43.563567 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c843f24-0220-4063-8d6d-d1f84196d9b2-kube-api-access-8m7bm\" (UniqueName: \"kubernetes.io/projected/8c843f24-0220-4063-8d6d-d1f84196d9b2-kube-api-access-8m7bm\") pod \"8c843f24-0220-4063-8d6d-d1f84196d9b2\" (UID: \"8c843f24-0220-4063-8d6d-d1f84196d9b2\") " Oct 30 13:03:43.563792 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:43.563774 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c843f24-0220-4063-8d6d-d1f84196d9b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c843f24-0220-4063-8d6d-d1f84196d9b2-openshift-service-ca\") pod \"8c843f24-0220-4063-8d6d-d1f84196d9b2\" (UID: \"8c843f24-0220-4063-8d6d-d1f84196d9b2\") " Oct 30 13:03:43.563868 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:43.563819 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c843f24-0220-4063-8d6d-d1f84196d9b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c843f24-0220-4063-8d6d-d1f84196d9b2-maas-api-tls\") pod \"8c843f24-0220-4063-8d6d-d1f84196d9b2\" (UID: \"8c843f24-0220-4063-8d6d-d1f84196d9b2\") " Oct 30 13:03:43.564035 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:43.564020 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c843f24-0220-4063-8d6d-d1f84196d9b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c843f24-0220-4063-8d6d-d1f84196d9b2-metrics-certs\") pod \"8c843f24-0220-4063-8d6d-d1f84196d9b2\" (UID: \"8c843f24-0220-4063-8d6d-d1f84196d9b2\") " Oct 30 13:03:43.564179 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:43.564152 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c843f24-0220-4063-8d6d-d1f84196d9b2-openshift-service-ca" pod "8c843f24-0220-4063-8d6d-d1f84196d9b2" (UID: "8c843f24-0220-4063-8d6d-d1f84196d9b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:03:43.564587 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:43.564562 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c843f24-0220-4063-8d6d-d1f84196d9b2-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:43.565985 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:43.565954 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c843f24-0220-4063-8d6d-d1f84196d9b2-kube-api-access-8m7bm" pod "8c843f24-0220-4063-8d6d-d1f84196d9b2" (UID: "8c843f24-0220-4063-8d6d-d1f84196d9b2"). InnerVolumeSpecName "kube-api-access-8m7bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:43.566089 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:43.566016 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c843f24-0220-4063-8d6d-d1f84196d9b2-maas-api-tls" pod "8c843f24-0220-4063-8d6d-d1f84196d9b2" (UID: "8c843f24-0220-4063-8d6d-d1f84196d9b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:03:43.566201 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:43.566185 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c843f24-0220-4063-8d6d-d1f84196d9b2-metrics-certs" pod "8c843f24-0220-4063-8d6d-d1f84196d9b2" (UID: "8c843f24-0220-4063-8d6d-d1f84196d9b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:03:43.667051 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:43.666961 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c843f24-0220-4063-8d6d-d1f84196d9b2-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:43.667051 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:43.667003 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m7bm\" (UniqueName: \"kubernetes.io/projected/8c843f24-0220-4063-8d6d-d1f84196d9b2-kube-api-access-8m7bm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:43.667051 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:43.667020 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c843f24-0220-4063-8d6d-d1f84196d9b2-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:43.735226 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:43.735190 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:43.857066 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:43.857039 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:44.016970 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:44.016888 4413 generic.go:356] "Generic (PLEG): container finished" podID="8c843f24-0220-4063-8d6d-d1f84196d9b2" containerID="a43fe32318610c421d56f6785362614143c4bce112bb7e64cc52422e85efecf4" exitCode=0 Oct 30 13:03:44.016970 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:44.016925 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" Oct 30 13:03:44.017203 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:44.016967 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" event={"ID":"8c843f24-0220-4063-8d6d-d1f84196d9b2","Type":"ContainerDied","Data":"a43fe32318610c421d56f6785362614143c4bce112bb7e64cc52422e85efecf4"} Oct 30 13:03:44.017203 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:44.017012 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479" event={"ID":"8c843f24-0220-4063-8d6d-d1f84196d9b2","Type":"ContainerDied","Data":"71b063a271178352c9a518d027d20b6580f6e795527a86afecafc47450434870"} Oct 30 13:03:44.017203 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:44.017037 4413 scope.go:122] "RemoveContainer" containerID="a43fe32318610c421d56f6785362614143c4bce112bb7e64cc52422e85efecf4" Oct 30 13:03:44.028882 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:44.028862 4413 scope.go:122] "RemoveContainer" containerID="a43fe32318610c421d56f6785362614143c4bce112bb7e64cc52422e85efecf4" Oct 30 13:03:44.029142 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:03:44.029123 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43fe32318610c421d56f6785362614143c4bce112bb7e64cc52422e85efecf4\": container with ID starting with a43fe32318610c421d56f6785362614143c4bce112bb7e64cc52422e85efecf4 not found: ID does not exist" containerID="a43fe32318610c421d56f6785362614143c4bce112bb7e64cc52422e85efecf4" Oct 30 13:03:44.029227 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:44.029150 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43fe32318610c421d56f6785362614143c4bce112bb7e64cc52422e85efecf4"} err="failed to get container status \"a43fe32318610c421d56f6785362614143c4bce112bb7e64cc52422e85efecf4\": rpc error: code = NotFound desc = could not find container \"a43fe32318610c421d56f6785362614143c4bce112bb7e64cc52422e85efecf4\": container with ID starting with a43fe32318610c421d56f6785362614143c4bce112bb7e64cc52422e85efecf4 not found: ID does not exist" Oct 30 13:03:44.069680 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:44.069649 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479"] Oct 30 13:03:44.081110 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:44.081082 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecb3aa-5ff6f585df-k6479"] Oct 30 13:03:44.353380 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:44.353287 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:45.022454 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:45.022422 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c843f24-0220-4063-8d6d-d1f84196d9b2" path="/var/lib/kubelet/pods/8c843f24-0220-4063-8d6d-d1f84196d9b2/volumes" Oct 30 13:03:45.563832 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:45.563788 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:47.656189 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:47.656151 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:49.286535 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:49.286456 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:49.584821 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:49.584732 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" podUID="fe249b4d-be8a-4135-acfc-568903715ef6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:03:49.584982 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:49.584857 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" podUID="ed73650f-c307-4db1-80c1-9b129dd488c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:03:50.960358 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:50.960331 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" Oct 30 13:03:50.974409 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:50.974370 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:51.004984 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.004948 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" Oct 30 13:03:51.068866 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.068830 4413 generic.go:356] "Generic (PLEG): container finished" podID="fe249b4d-be8a-4135-acfc-568903715ef6" containerID="e25e452e6225db4a7b91f13763d2716e412a6bcd57f3abe66605dccddcdbe2c6" exitCode=0 Oct 30 13:03:51.069037 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.068877 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" Oct 30 13:03:51.069037 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.068910 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" event={"ID":"fe249b4d-be8a-4135-acfc-568903715ef6","Type":"ContainerDied","Data":"e25e452e6225db4a7b91f13763d2716e412a6bcd57f3abe66605dccddcdbe2c6"} Oct 30 13:03:51.069037 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.068973 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m" event={"ID":"fe249b4d-be8a-4135-acfc-568903715ef6","Type":"ContainerDied","Data":"74940f8fa45f40af8a1b7e065e635ecd37c648bf5bc8a6e0ef278bf32a62aacd"} Oct 30 13:03:51.069037 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.069007 4413 scope.go:122] "RemoveContainer" containerID="e25e452e6225db4a7b91f13763d2716e412a6bcd57f3abe66605dccddcdbe2c6" Oct 30 13:03:51.071071 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.071046 4413 generic.go:356] "Generic (PLEG): container finished" podID="ed73650f-c307-4db1-80c1-9b129dd488c4" containerID="48ade5fc144d34274f82e39b54d8a78aade6d47cdd6339a946beb984ff3dbcee" exitCode=0 Oct 30 13:03:51.071202 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.071095 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" Oct 30 13:03:51.071202 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.071134 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" event={"ID":"ed73650f-c307-4db1-80c1-9b129dd488c4","Type":"ContainerDied","Data":"48ade5fc144d34274f82e39b54d8a78aade6d47cdd6339a946beb984ff3dbcee"} Oct 30 13:03:51.071202 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.071180 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7" event={"ID":"ed73650f-c307-4db1-80c1-9b129dd488c4","Type":"ContainerDied","Data":"8203ac9fbf82c0259ab7ecf0ea096a3d2541902fc0c4e6b5f0e41f7e59ad45c1"} Oct 30 13:03:51.071696 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.071674 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe249b4d-be8a-4135-acfc-568903715ef6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe249b4d-be8a-4135-acfc-568903715ef6-plugins-config-volume\") pod \"fe249b4d-be8a-4135-acfc-568903715ef6\" (UID: \"fe249b4d-be8a-4135-acfc-568903715ef6\") " Oct 30 13:03:51.071820 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.071802 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed73650f-c307-4db1-80c1-9b129dd488c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed73650f-c307-4db1-80c1-9b129dd488c4-plugins-config-volume\") pod \"ed73650f-c307-4db1-80c1-9b129dd488c4\" (UID: \"ed73650f-c307-4db1-80c1-9b129dd488c4\") " Oct 30 13:03:51.071947 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.071875 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe249b4d-be8a-4135-acfc-568903715ef6-kube-api-access-sfcmq\" (UniqueName: \"kubernetes.io/projected/fe249b4d-be8a-4135-acfc-568903715ef6-kube-api-access-sfcmq\") pod \"fe249b4d-be8a-4135-acfc-568903715ef6\" (UID: \"fe249b4d-be8a-4135-acfc-568903715ef6\") " Oct 30 13:03:51.072108 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.072089 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed73650f-c307-4db1-80c1-9b129dd488c4-kube-api-access-wg2lj\" (UniqueName: \"kubernetes.io/projected/ed73650f-c307-4db1-80c1-9b129dd488c4-kube-api-access-wg2lj\") pod \"ed73650f-c307-4db1-80c1-9b129dd488c4\" (UID: \"ed73650f-c307-4db1-80c1-9b129dd488c4\") " Oct 30 13:03:51.072232 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.072207 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed73650f-c307-4db1-80c1-9b129dd488c4-plugins-config-volume" pod "ed73650f-c307-4db1-80c1-9b129dd488c4" (UID: "ed73650f-c307-4db1-80c1-9b129dd488c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:03:51.072443 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.072423 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe249b4d-be8a-4135-acfc-568903715ef6-plugins-config-volume" pod "fe249b4d-be8a-4135-acfc-568903715ef6" (UID: "fe249b4d-be8a-4135-acfc-568903715ef6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:03:51.073557 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.073522 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe249b4d-be8a-4135-acfc-568903715ef6-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:51.073557 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.073550 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed73650f-c307-4db1-80c1-9b129dd488c4-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:51.074908 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.074885 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe249b4d-be8a-4135-acfc-568903715ef6-kube-api-access-sfcmq" pod "fe249b4d-be8a-4135-acfc-568903715ef6" (UID: "fe249b4d-be8a-4135-acfc-568903715ef6"). InnerVolumeSpecName "kube-api-access-sfcmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:51.077018 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.076995 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed73650f-c307-4db1-80c1-9b129dd488c4-kube-api-access-wg2lj" pod "ed73650f-c307-4db1-80c1-9b129dd488c4" (UID: "ed73650f-c307-4db1-80c1-9b129dd488c4"). InnerVolumeSpecName "kube-api-access-wg2lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:03:51.083690 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.083667 4413 scope.go:122] "RemoveContainer" containerID="e25e452e6225db4a7b91f13763d2716e412a6bcd57f3abe66605dccddcdbe2c6" Oct 30 13:03:51.084003 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:03:51.083981 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25e452e6225db4a7b91f13763d2716e412a6bcd57f3abe66605dccddcdbe2c6\": container with ID starting with e25e452e6225db4a7b91f13763d2716e412a6bcd57f3abe66605dccddcdbe2c6 not found: ID does not exist" containerID="e25e452e6225db4a7b91f13763d2716e412a6bcd57f3abe66605dccddcdbe2c6" Oct 30 13:03:51.084096 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.084011 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25e452e6225db4a7b91f13763d2716e412a6bcd57f3abe66605dccddcdbe2c6"} err="failed to get container status \"e25e452e6225db4a7b91f13763d2716e412a6bcd57f3abe66605dccddcdbe2c6\": rpc error: code = NotFound desc = could not find container \"e25e452e6225db4a7b91f13763d2716e412a6bcd57f3abe66605dccddcdbe2c6\": container with ID starting with e25e452e6225db4a7b91f13763d2716e412a6bcd57f3abe66605dccddcdbe2c6 not found: ID does not exist" Oct 30 13:03:51.084096 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.084029 4413 scope.go:122] "RemoveContainer" containerID="48ade5fc144d34274f82e39b54d8a78aade6d47cdd6339a946beb984ff3dbcee" Oct 30 13:03:51.097846 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.097822 4413 scope.go:122] "RemoveContainer" containerID="48ade5fc144d34274f82e39b54d8a78aade6d47cdd6339a946beb984ff3dbcee" Oct 30 13:03:51.112504 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:03:51.098304 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ade5fc144d34274f82e39b54d8a78aade6d47cdd6339a946beb984ff3dbcee\": container with ID starting with 48ade5fc144d34274f82e39b54d8a78aade6d47cdd6339a946beb984ff3dbcee not found: ID does not exist" containerID="48ade5fc144d34274f82e39b54d8a78aade6d47cdd6339a946beb984ff3dbcee" Oct 30 13:03:51.112504 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.098365 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ade5fc144d34274f82e39b54d8a78aade6d47cdd6339a946beb984ff3dbcee"} err="failed to get container status \"48ade5fc144d34274f82e39b54d8a78aade6d47cdd6339a946beb984ff3dbcee\": rpc error: code = NotFound desc = could not find container \"48ade5fc144d34274f82e39b54d8a78aade6d47cdd6339a946beb984ff3dbcee\": container with ID starting with 48ade5fc144d34274f82e39b54d8a78aade6d47cdd6339a946beb984ff3dbcee not found: ID does not exist" Oct 30 13:03:51.112504 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.109802 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:51.175414 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.175336 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg2lj\" (UniqueName: \"kubernetes.io/projected/ed73650f-c307-4db1-80c1-9b129dd488c4-kube-api-access-wg2lj\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:51.175414 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.175361 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfcmq\" (UniqueName: \"kubernetes.io/projected/fe249b4d-be8a-4135-acfc-568903715ef6-kube-api-access-sfcmq\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:03:51.407903 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:03:51.407869 4413 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe249b4d_be8a_4135_acfc_568903715ef6.slice/crio-74940f8fa45f40af8a1b7e065e635ecd37c648bf5bc8a6e0ef278bf32a62aacd\": RecentStats: unable to find data in memory cache]" Oct 30 13:03:51.429144 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.429045 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m"] Oct 30 13:03:51.435048 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.435024 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e9f184-76fc4c8bbd-t4d9m"] Oct 30 13:03:51.534274 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.530460 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:51.534493 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.534397 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7"] Oct 30 13:03:51.547276 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:51.546902 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e9f184-64d65589f-s7qf7"] Oct 30 13:03:53.022016 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:53.021981 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed73650f-c307-4db1-80c1-9b129dd488c4" path="/var/lib/kubelet/pods/ed73650f-c307-4db1-80c1-9b129dd488c4/volumes" Oct 30 13:03:53.022673 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:53.022652 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe249b4d-be8a-4135-acfc-568903715ef6" path="/var/lib/kubelet/pods/fe249b4d-be8a-4135-acfc-568903715ef6/volumes" Oct 30 13:03:53.056027 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:53.055998 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:54.787512 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:54.787479 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:56.562526 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:56.562492 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:03:58.530885 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:03:58.530852 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:00.159507 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:00.159473 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:01.872757 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:01.872721 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:02.365435 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:02.365399 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7"] Oct 30 13:04:02.401135 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:02.401101 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk"] Oct 30 13:04:02.685737 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:02.685704 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:02.867461 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:02.867424 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:03.000506 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:03.000425 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:03.343360 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:03.343269 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:03.548512 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:03.548481 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:03.678107 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:03.678068 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:05.386020 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:05.385989 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:05.819046 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:04:05.818967 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:04:06.989754 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:06.989719 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:07.366103 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:07.366011 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" podUID="2232e7b3-95ec-4866-81a9-30a396bf9d96" containerName="payload-processing" containerID="cri-o://19d4d13dd4df519ef05bd56f60b6da4c13e1499a81ab9406a9ba96c7477ca693" gracePeriod=25 Oct 30 13:04:07.401977 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:07.401931 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" podUID="272a58bb-d7d5-4c02-a278-1dd57cdc75bf" containerName="payload-pre-processing" containerID="cri-o://b7cdc327b48b65efd776da2fb70673c974754f0b6f198236741be6b58bfd0b65" gracePeriod=25 Oct 30 13:04:08.424441 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:08.424407 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt"] Oct 30 13:04:08.424928 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:08.424736 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" podUID="22d9217c-a901-46fa-ba61-21bc01e658cb" containerName="maas-api" containerID="cri-o://1af290ebfa8ddac13fa6ff344814dca42e9b79b499523279286a7035b13fbb53" gracePeriod=30 Oct 30 13:04:08.495925 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:08.495889 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:08.742155 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:08.742131 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" Oct 30 13:04:08.758956 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:08.758927 4413 scope.go:122] "RemoveContainer" containerID="1af290ebfa8ddac13fa6ff344814dca42e9b79b499523279286a7035b13fbb53" Oct 30 13:04:08.869015 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:08.868978 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22d9217c-a901-46fa-ba61-21bc01e658cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/22d9217c-a901-46fa-ba61-21bc01e658cb-metrics-certs\") pod \"22d9217c-a901-46fa-ba61-21bc01e658cb\" (UID: \"22d9217c-a901-46fa-ba61-21bc01e658cb\") " Oct 30 13:04:08.869222 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:08.869204 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22d9217c-a901-46fa-ba61-21bc01e658cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d9217c-a901-46fa-ba61-21bc01e658cb-openshift-service-ca\") pod \"22d9217c-a901-46fa-ba61-21bc01e658cb\" (UID: \"22d9217c-a901-46fa-ba61-21bc01e658cb\") " Oct 30 13:04:08.869454 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:08.869438 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d9217c-a901-46fa-ba61-21bc01e658cb-kube-api-access-h947z\" (UniqueName: \"kubernetes.io/projected/22d9217c-a901-46fa-ba61-21bc01e658cb-kube-api-access-h947z\") pod \"22d9217c-a901-46fa-ba61-21bc01e658cb\" (UID: \"22d9217c-a901-46fa-ba61-21bc01e658cb\") " Oct 30 13:04:08.869649 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:08.869634 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22d9217c-a901-46fa-ba61-21bc01e658cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d9217c-a901-46fa-ba61-21bc01e658cb-maas-api-tls\") pod \"22d9217c-a901-46fa-ba61-21bc01e658cb\" (UID: \"22d9217c-a901-46fa-ba61-21bc01e658cb\") " Oct 30 13:04:08.870725 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:08.870700 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d9217c-a901-46fa-ba61-21bc01e658cb-openshift-service-ca" pod "22d9217c-a901-46fa-ba61-21bc01e658cb" (UID: "22d9217c-a901-46fa-ba61-21bc01e658cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:08.876623 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:08.876602 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22d9217c-a901-46fa-ba61-21bc01e658cb-metrics-certs" pod "22d9217c-a901-46fa-ba61-21bc01e658cb" (UID: "22d9217c-a901-46fa-ba61-21bc01e658cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:08.881709 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:08.881675 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22d9217c-a901-46fa-ba61-21bc01e658cb-maas-api-tls" pod "22d9217c-a901-46fa-ba61-21bc01e658cb" (UID: "22d9217c-a901-46fa-ba61-21bc01e658cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:08.883904 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:08.883869 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d9217c-a901-46fa-ba61-21bc01e658cb-kube-api-access-h947z" pod "22d9217c-a901-46fa-ba61-21bc01e658cb" (UID: "22d9217c-a901-46fa-ba61-21bc01e658cb"). InnerVolumeSpecName "kube-api-access-h947z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:08.972004 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:08.971936 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22d9217c-a901-46fa-ba61-21bc01e658cb-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:08.972004 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:08.971964 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h947z\" (UniqueName: \"kubernetes.io/projected/22d9217c-a901-46fa-ba61-21bc01e658cb-kube-api-access-h947z\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:08.972004 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:08.971980 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22d9217c-a901-46fa-ba61-21bc01e658cb-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:08.972004 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:08.971989 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22d9217c-a901-46fa-ba61-21bc01e658cb-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:09.028379 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:09.026582 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:09.152081 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:09.152047 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:09.226542 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:09.226404 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" event={"ID":"22d9217c-a901-46fa-ba61-21bc01e658cb","Type":"ContainerDied","Data":"1af290ebfa8ddac13fa6ff344814dca42e9b79b499523279286a7035b13fbb53"} Oct 30 13:04:09.226542 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:09.226478 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" event={"ID":"22d9217c-a901-46fa-ba61-21bc01e658cb","Type":"ContainerDied","Data":"19f68d5dc0f2bc1b38ba7f16af6ec2d5e0f5de2db1b6b6be5ca4c9ddc6631377"} Oct 30 13:04:09.226801 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:09.226558 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt" Oct 30 13:04:09.283668 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:09.283637 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:09.307528 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:09.307489 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt"] Oct 30 13:04:09.312468 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:09.312447 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e9f184-65774d7fd9-52xnt"] Oct 30 13:04:09.639879 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:09.639849 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:11.023260 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:11.023216 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22d9217c-a901-46fa-ba61-21bc01e658cb" path="/var/lib/kubelet/pods/22d9217c-a901-46fa-ba61-21bc01e658cb/volumes" Oct 30 13:04:11.040968 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:11.040943 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:13.054283 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:13.054236 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:13.691169 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:13.691134 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" podUID="272a58bb-d7d5-4c02-a278-1dd57cdc75bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:04:13.691341 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:13.691172 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" podUID="2232e7b3-95ec-4866-81a9-30a396bf9d96" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:04:14.795218 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:14.795186 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:15.718240 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:15.718210 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:16.725667 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:16.725633 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:18.034672 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:18.034641 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:18.155143 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:18.155107 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:18.370373 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:18.370299 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:18.571234 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:18.571201 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:20.488831 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:20.488794 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:21.993027 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:21.992996 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:22.594305 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:22.594270 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:22.742933 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:22.742899 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:22.858945 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:22.858868 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:23.226066 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:23.226031 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:23.690905 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:23.690863 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" podUID="272a58bb-d7d5-4c02-a278-1dd57cdc75bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:04:23.691090 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:23.690870 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" podUID="2232e7b3-95ec-4866-81a9-30a396bf9d96" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:04:24.717568 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:24.717537 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:26.463355 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:26.463312 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:27.414015 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:27.413980 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct"] Oct 30 13:04:27.497030 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:27.496995 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:27.738143 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:27.738107 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" Oct 30 13:04:27.779686 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:27.779666 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" Oct 30 13:04:27.829083 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:27.829037 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/272a58bb-d7d5-4c02-a278-1dd57cdc75bf-kube-api-access-brmgp\" (UniqueName: \"kubernetes.io/projected/272a58bb-d7d5-4c02-a278-1dd57cdc75bf-kube-api-access-brmgp\") pod \"272a58bb-d7d5-4c02-a278-1dd57cdc75bf\" (UID: \"272a58bb-d7d5-4c02-a278-1dd57cdc75bf\") " Oct 30 13:04:27.829321 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:27.829153 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2232e7b3-95ec-4866-81a9-30a396bf9d96-kube-api-access-n6qf4\" (UniqueName: \"kubernetes.io/projected/2232e7b3-95ec-4866-81a9-30a396bf9d96-kube-api-access-n6qf4\") pod \"2232e7b3-95ec-4866-81a9-30a396bf9d96\" (UID: \"2232e7b3-95ec-4866-81a9-30a396bf9d96\") " Oct 30 13:04:27.829321 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:27.829297 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/272a58bb-d7d5-4c02-a278-1dd57cdc75bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272a58bb-d7d5-4c02-a278-1dd57cdc75bf-plugins-config-volume\") pod \"272a58bb-d7d5-4c02-a278-1dd57cdc75bf\" (UID: \"272a58bb-d7d5-4c02-a278-1dd57cdc75bf\") " Oct 30 13:04:27.829555 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:27.829341 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2232e7b3-95ec-4866-81a9-30a396bf9d96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2232e7b3-95ec-4866-81a9-30a396bf9d96-plugins-config-volume\") pod \"2232e7b3-95ec-4866-81a9-30a396bf9d96\" (UID: \"2232e7b3-95ec-4866-81a9-30a396bf9d96\") " Oct 30 13:04:27.830000 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:27.829977 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272a58bb-d7d5-4c02-a278-1dd57cdc75bf-plugins-config-volume" pod "272a58bb-d7d5-4c02-a278-1dd57cdc75bf" (UID: "272a58bb-d7d5-4c02-a278-1dd57cdc75bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:27.830792 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:27.830764 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2232e7b3-95ec-4866-81a9-30a396bf9d96-plugins-config-volume" pod "2232e7b3-95ec-4866-81a9-30a396bf9d96" (UID: "2232e7b3-95ec-4866-81a9-30a396bf9d96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:27.836160 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:27.836125 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2232e7b3-95ec-4866-81a9-30a396bf9d96-kube-api-access-n6qf4" pod "2232e7b3-95ec-4866-81a9-30a396bf9d96" (UID: "2232e7b3-95ec-4866-81a9-30a396bf9d96"). InnerVolumeSpecName "kube-api-access-n6qf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:27.837273 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:27.837225 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272a58bb-d7d5-4c02-a278-1dd57cdc75bf-kube-api-access-brmgp" pod "272a58bb-d7d5-4c02-a278-1dd57cdc75bf" (UID: "272a58bb-d7d5-4c02-a278-1dd57cdc75bf"). InnerVolumeSpecName "kube-api-access-brmgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:27.931571 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:27.931494 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6qf4\" (UniqueName: \"kubernetes.io/projected/2232e7b3-95ec-4866-81a9-30a396bf9d96-kube-api-access-n6qf4\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:27.931571 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:27.931523 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272a58bb-d7d5-4c02-a278-1dd57cdc75bf-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:27.931571 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:27.931533 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2232e7b3-95ec-4866-81a9-30a396bf9d96-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:27.931571 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:27.931541 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brmgp\" (UniqueName: \"kubernetes.io/projected/272a58bb-d7d5-4c02-a278-1dd57cdc75bf-kube-api-access-brmgp\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:28.262071 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.261992 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:28.374838 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.374806 4413 generic.go:356] "Generic (PLEG): container finished" podID="2232e7b3-95ec-4866-81a9-30a396bf9d96" containerID="19d4d13dd4df519ef05bd56f60b6da4c13e1499a81ab9406a9ba96c7477ca693" exitCode=0 Oct 30 13:04:28.374981 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.374858 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" Oct 30 13:04:28.374981 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.374877 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" event={"ID":"2232e7b3-95ec-4866-81a9-30a396bf9d96","Type":"ContainerDied","Data":"19d4d13dd4df519ef05bd56f60b6da4c13e1499a81ab9406a9ba96c7477ca693"} Oct 30 13:04:28.374981 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.374936 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7" event={"ID":"2232e7b3-95ec-4866-81a9-30a396bf9d96","Type":"ContainerDied","Data":"07e1c44fc4c11e4766903b16c392ce54be1078f4871b77b2c62cb5ee39e71a7d"} Oct 30 13:04:28.374981 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.374968 4413 scope.go:122] "RemoveContainer" containerID="19d4d13dd4df519ef05bd56f60b6da4c13e1499a81ab9406a9ba96c7477ca693" Oct 30 13:04:28.377315 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.377286 4413 generic.go:356] "Generic (PLEG): container finished" podID="272a58bb-d7d5-4c02-a278-1dd57cdc75bf" containerID="b7cdc327b48b65efd776da2fb70673c974754f0b6f198236741be6b58bfd0b65" exitCode=0 Oct 30 13:04:28.377461 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.377331 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" Oct 30 13:04:28.377461 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.377347 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" event={"ID":"272a58bb-d7d5-4c02-a278-1dd57cdc75bf","Type":"ContainerDied","Data":"b7cdc327b48b65efd776da2fb70673c974754f0b6f198236741be6b58bfd0b65"} Oct 30 13:04:28.377461 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.377403 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk" event={"ID":"272a58bb-d7d5-4c02-a278-1dd57cdc75bf","Type":"ContainerDied","Data":"2c8b125fbb6c3b2fc0058db0ac680cdaa1c10a3c7d175cf061edde00bf108a34"} Oct 30 13:04:28.396449 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.396431 4413 scope.go:122] "RemoveContainer" containerID="19d4d13dd4df519ef05bd56f60b6da4c13e1499a81ab9406a9ba96c7477ca693" Oct 30 13:04:28.396746 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:04:28.396730 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19d4d13dd4df519ef05bd56f60b6da4c13e1499a81ab9406a9ba96c7477ca693\": container with ID starting with 19d4d13dd4df519ef05bd56f60b6da4c13e1499a81ab9406a9ba96c7477ca693 not found: ID does not exist" containerID="19d4d13dd4df519ef05bd56f60b6da4c13e1499a81ab9406a9ba96c7477ca693" Oct 30 13:04:28.396809 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.396755 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19d4d13dd4df519ef05bd56f60b6da4c13e1499a81ab9406a9ba96c7477ca693"} err="failed to get container status \"19d4d13dd4df519ef05bd56f60b6da4c13e1499a81ab9406a9ba96c7477ca693\": rpc error: code = NotFound desc = could not find container \"19d4d13dd4df519ef05bd56f60b6da4c13e1499a81ab9406a9ba96c7477ca693\": container with ID starting with 19d4d13dd4df519ef05bd56f60b6da4c13e1499a81ab9406a9ba96c7477ca693 not found: ID does not exist" Oct 30 13:04:28.396809 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.396772 4413 scope.go:122] "RemoveContainer" containerID="b7cdc327b48b65efd776da2fb70673c974754f0b6f198236741be6b58bfd0b65" Oct 30 13:04:28.406691 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.406671 4413 scope.go:122] "RemoveContainer" containerID="b7cdc327b48b65efd776da2fb70673c974754f0b6f198236741be6b58bfd0b65" Oct 30 13:04:28.406944 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:04:28.406925 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7cdc327b48b65efd776da2fb70673c974754f0b6f198236741be6b58bfd0b65\": container with ID starting with b7cdc327b48b65efd776da2fb70673c974754f0b6f198236741be6b58bfd0b65 not found: ID does not exist" containerID="b7cdc327b48b65efd776da2fb70673c974754f0b6f198236741be6b58bfd0b65" Oct 30 13:04:28.407007 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.406956 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7cdc327b48b65efd776da2fb70673c974754f0b6f198236741be6b58bfd0b65"} err="failed to get container status \"b7cdc327b48b65efd776da2fb70673c974754f0b6f198236741be6b58bfd0b65\": rpc error: code = NotFound desc = could not find container \"b7cdc327b48b65efd776da2fb70673c974754f0b6f198236741be6b58bfd0b65\": container with ID starting with b7cdc327b48b65efd776da2fb70673c974754f0b6f198236741be6b58bfd0b65 not found: ID does not exist" Oct 30 13:04:28.444032 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.444000 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk"] Oct 30 13:04:28.455507 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.455478 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0bda67-767cdbf49f-v2thk"] Oct 30 13:04:28.489623 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.489581 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7"] Oct 30 13:04:28.498684 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:28.498648 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0bda67-64b9d5945d-qcqh7"] Oct 30 13:04:29.021857 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:29.021827 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2232e7b3-95ec-4866-81a9-30a396bf9d96" path="/var/lib/kubelet/pods/2232e7b3-95ec-4866-81a9-30a396bf9d96/volumes" Oct 30 13:04:29.022233 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:29.022218 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="272a58bb-d7d5-4c02-a278-1dd57cdc75bf" path="/var/lib/kubelet/pods/272a58bb-d7d5-4c02-a278-1dd57cdc75bf/volumes" Oct 30 13:04:29.349447 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:29.349365 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:29.465577 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:29.465543 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:30.032424 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:30.032391 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:30.941929 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:30.941898 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:31.100038 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:31.100003 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:31.289758 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:31.289682 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8"] Oct 30 13:04:31.289952 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:31.289928 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" podUID="6435f44e-980e-4166-868b-ff94fa2010b9" containerName="istio-proxy" containerID="cri-o://e870bee964d94c7a593084b16f12abb8b855bd03eb8cb5f101b850720897f9f9" gracePeriod=30 Oct 30 13:04:31.334594 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:31.334554 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:31.925619 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:31.925556 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:33.343668 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:33.343630 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm"] Oct 30 13:04:33.344126 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:33.343874 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" podUID="747e931b-c949-4428-afe2-16da8de97a9a" containerName="maas-api" containerID="cri-o://6767f73c34057a9c1dadfd1073c88a4720fda9d1cdc8ce599fee507745703c09" gracePeriod=30 Oct 30 13:04:33.446346 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:33.446317 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:33.610936 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:33.610866 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" Oct 30 13:04:33.720581 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:33.720538 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/747e931b-c949-4428-afe2-16da8de97a9a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/747e931b-c949-4428-afe2-16da8de97a9a-maas-api-tls\") pod \"747e931b-c949-4428-afe2-16da8de97a9a\" (UID: \"747e931b-c949-4428-afe2-16da8de97a9a\") " Oct 30 13:04:33.720805 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:33.720665 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747e931b-c949-4428-afe2-16da8de97a9a-kube-api-access-bmmc5\" (UniqueName: \"kubernetes.io/projected/747e931b-c949-4428-afe2-16da8de97a9a-kube-api-access-bmmc5\") pod \"747e931b-c949-4428-afe2-16da8de97a9a\" (UID: \"747e931b-c949-4428-afe2-16da8de97a9a\") " Oct 30 13:04:33.720805 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:33.720786 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/747e931b-c949-4428-afe2-16da8de97a9a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/747e931b-c949-4428-afe2-16da8de97a9a-openshift-service-ca\") pod \"747e931b-c949-4428-afe2-16da8de97a9a\" (UID: \"747e931b-c949-4428-afe2-16da8de97a9a\") " Oct 30 13:04:33.720928 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:33.720861 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/747e931b-c949-4428-afe2-16da8de97a9a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/747e931b-c949-4428-afe2-16da8de97a9a-metrics-certs\") pod \"747e931b-c949-4428-afe2-16da8de97a9a\" (UID: \"747e931b-c949-4428-afe2-16da8de97a9a\") " Oct 30 13:04:33.721219 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:33.721188 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747e931b-c949-4428-afe2-16da8de97a9a-openshift-service-ca" pod "747e931b-c949-4428-afe2-16da8de97a9a" (UID: "747e931b-c949-4428-afe2-16da8de97a9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:33.721585 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:33.721499 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/747e931b-c949-4428-afe2-16da8de97a9a-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:33.723004 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:33.722964 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/747e931b-c949-4428-afe2-16da8de97a9a-maas-api-tls" pod "747e931b-c949-4428-afe2-16da8de97a9a" (UID: "747e931b-c949-4428-afe2-16da8de97a9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:33.723004 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:33.722971 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747e931b-c949-4428-afe2-16da8de97a9a-kube-api-access-bmmc5" pod "747e931b-c949-4428-afe2-16da8de97a9a" (UID: "747e931b-c949-4428-afe2-16da8de97a9a"). InnerVolumeSpecName "kube-api-access-bmmc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:33.723165 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:33.723069 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/747e931b-c949-4428-afe2-16da8de97a9a-metrics-certs" pod "747e931b-c949-4428-afe2-16da8de97a9a" (UID: "747e931b-c949-4428-afe2-16da8de97a9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:33.823728 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:33.823692 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747e931b-c949-4428-afe2-16da8de97a9a-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:33.823728 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:33.823731 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/747e931b-c949-4428-afe2-16da8de97a9a-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:33.824033 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:33.823746 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmmc5\" (UniqueName: \"kubernetes.io/projected/747e931b-c949-4428-afe2-16da8de97a9a-kube-api-access-bmmc5\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:34.083639 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:34.083593 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:34.215515 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:34.215484 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:34.431114 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:34.431079 4413 generic.go:356] "Generic (PLEG): container finished" podID="747e931b-c949-4428-afe2-16da8de97a9a" containerID="6767f73c34057a9c1dadfd1073c88a4720fda9d1cdc8ce599fee507745703c09" exitCode=0 Oct 30 13:04:34.431652 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:34.431144 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" Oct 30 13:04:34.431652 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:34.431169 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" event={"ID":"747e931b-c949-4428-afe2-16da8de97a9a","Type":"ContainerDied","Data":"6767f73c34057a9c1dadfd1073c88a4720fda9d1cdc8ce599fee507745703c09"} Oct 30 13:04:34.431652 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:34.431228 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm" event={"ID":"747e931b-c949-4428-afe2-16da8de97a9a","Type":"ContainerDied","Data":"79af6a9063a0321877f26d926be1cb120503fcf03703a305c824658f903e766c"} Oct 30 13:04:34.431652 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:34.431244 4413 scope.go:122] "RemoveContainer" containerID="6767f73c34057a9c1dadfd1073c88a4720fda9d1cdc8ce599fee507745703c09" Oct 30 13:04:34.443367 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:34.443351 4413 scope.go:122] "RemoveContainer" containerID="6767f73c34057a9c1dadfd1073c88a4720fda9d1cdc8ce599fee507745703c09" Oct 30 13:04:34.443618 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:04:34.443600 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6767f73c34057a9c1dadfd1073c88a4720fda9d1cdc8ce599fee507745703c09\": container with ID starting with 6767f73c34057a9c1dadfd1073c88a4720fda9d1cdc8ce599fee507745703c09 not found: ID does not exist" containerID="6767f73c34057a9c1dadfd1073c88a4720fda9d1cdc8ce599fee507745703c09" Oct 30 13:04:34.443668 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:34.443626 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6767f73c34057a9c1dadfd1073c88a4720fda9d1cdc8ce599fee507745703c09"} err="failed to get container status \"6767f73c34057a9c1dadfd1073c88a4720fda9d1cdc8ce599fee507745703c09\": rpc error: code = NotFound desc = could not find container \"6767f73c34057a9c1dadfd1073c88a4720fda9d1cdc8ce599fee507745703c09\": container with ID starting with 6767f73c34057a9c1dadfd1073c88a4720fda9d1cdc8ce599fee507745703c09 not found: ID does not exist" Oct 30 13:04:34.483423 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:34.483397 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm"] Oct 30 13:04:34.494978 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:34.494957 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6ad1da-5ccb8649c4-p9skm"] Oct 30 13:04:34.691883 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:34.691805 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:35.026610 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:35.026524 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="747e931b-c949-4428-afe2-16da8de97a9a" path="/var/lib/kubelet/pods/747e931b-c949-4428-afe2-16da8de97a9a/volumes" Oct 30 13:04:35.816036 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:04:35.815994 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:04:36.019749 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.019718 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:36.450539 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.450499 4413 generic.go:356] "Generic (PLEG): container finished" podID="6435f44e-980e-4166-868b-ff94fa2010b9" containerID="e870bee964d94c7a593084b16f12abb8b855bd03eb8cb5f101b850720897f9f9" exitCode=0 Oct 30 13:04:36.450725 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.450565 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" event={"ID":"6435f44e-980e-4166-868b-ff94fa2010b9","Type":"ContainerDied","Data":"e870bee964d94c7a593084b16f12abb8b855bd03eb8cb5f101b850720897f9f9"} Oct 30 13:04:36.450725 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.450633 4413 scope.go:122] "RemoveContainer" containerID="0b1529c128f54f2748ccf120adb919179f7481237fa77b6e906805a8aadc1ee9" Oct 30 13:04:36.586086 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.586061 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 13:04:36.663153 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.663120 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6435f44e-980e-4166-868b-ff94fa2010b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6435f44e-980e-4166-868b-ff94fa2010b9-istio-podinfo\") pod \"6435f44e-980e-4166-868b-ff94fa2010b9\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " Oct 30 13:04:36.663363 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.663188 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-workload-socket\") pod \"6435f44e-980e-4166-868b-ff94fa2010b9\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " Oct 30 13:04:36.663363 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.663292 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-credential-socket\") pod \"6435f44e-980e-4166-868b-ff94fa2010b9\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " Oct 30 13:04:36.663363 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.663330 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-istio-envoy\") pod \"6435f44e-980e-4166-868b-ff94fa2010b9\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " Oct 30 13:04:36.663625 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.663379 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6435f44e-980e-4166-868b-ff94fa2010b9-istio-token\" (UniqueName: \"kubernetes.io/projected/6435f44e-980e-4166-868b-ff94fa2010b9-istio-token\") pod \"6435f44e-980e-4166-868b-ff94fa2010b9\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " Oct 30 13:04:36.663625 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.663412 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6435f44e-980e-4166-868b-ff94fa2010b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6435f44e-980e-4166-868b-ff94fa2010b9-istiod-ca-cert\") pod \"6435f44e-980e-4166-868b-ff94fa2010b9\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " Oct 30 13:04:36.663625 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.663560 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6435f44e-980e-4166-868b-ff94fa2010b9-kube-api-access-vzrm2\" (UniqueName: \"kubernetes.io/projected/6435f44e-980e-4166-868b-ff94fa2010b9-kube-api-access-vzrm2\") pod \"6435f44e-980e-4166-868b-ff94fa2010b9\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " Oct 30 13:04:36.663800 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.663735 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-istio-data\") pod \"6435f44e-980e-4166-868b-ff94fa2010b9\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " Oct 30 13:04:36.663800 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.663788 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-workload-certs\") pod \"6435f44e-980e-4166-868b-ff94fa2010b9\" (UID: \"6435f44e-980e-4166-868b-ff94fa2010b9\") " Oct 30 13:04:36.663916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.663854 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-credential-socket" pod "6435f44e-980e-4166-868b-ff94fa2010b9" (UID: "6435f44e-980e-4166-868b-ff94fa2010b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:36.663916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.663866 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6435f44e-980e-4166-868b-ff94fa2010b9-istiod-ca-cert" pod "6435f44e-980e-4166-868b-ff94fa2010b9" (UID: "6435f44e-980e-4166-868b-ff94fa2010b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:36.663916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.663884 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-workload-socket" pod "6435f44e-980e-4166-868b-ff94fa2010b9" (UID: "6435f44e-980e-4166-868b-ff94fa2010b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:36.664145 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.664126 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-istio-data" pod "6435f44e-980e-4166-868b-ff94fa2010b9" (UID: "6435f44e-980e-4166-868b-ff94fa2010b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:36.664235 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.664146 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-workload-certs" pod "6435f44e-980e-4166-868b-ff94fa2010b9" (UID: "6435f44e-980e-4166-868b-ff94fa2010b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:36.664482 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.664453 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:36.664570 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.664483 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6435f44e-980e-4166-868b-ff94fa2010b9-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:36.664570 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.664499 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:36.664570 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.664511 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:36.664570 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.664526 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:36.667636 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.667592 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6435f44e-980e-4166-868b-ff94fa2010b9-istio-podinfo" pod "6435f44e-980e-4166-868b-ff94fa2010b9" (UID: "6435f44e-980e-4166-868b-ff94fa2010b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:04:36.667636 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.667622 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6435f44e-980e-4166-868b-ff94fa2010b9-istio-token" pod "6435f44e-980e-4166-868b-ff94fa2010b9" (UID: "6435f44e-980e-4166-868b-ff94fa2010b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:36.667808 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.667731 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-istio-envoy" pod "6435f44e-980e-4166-868b-ff94fa2010b9" (UID: "6435f44e-980e-4166-868b-ff94fa2010b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:36.668902 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.668874 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6435f44e-980e-4166-868b-ff94fa2010b9-kube-api-access-vzrm2" pod "6435f44e-980e-4166-868b-ff94fa2010b9" (UID: "6435f44e-980e-4166-868b-ff94fa2010b9"). InnerVolumeSpecName "kube-api-access-vzrm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:36.766309 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.766243 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6435f44e-980e-4166-868b-ff94fa2010b9-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:36.766309 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.766306 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6435f44e-980e-4166-868b-ff94fa2010b9-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:36.766309 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.766315 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzrm2\" (UniqueName: \"kubernetes.io/projected/6435f44e-980e-4166-868b-ff94fa2010b9-kube-api-access-vzrm2\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:36.766577 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:36.766324 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6435f44e-980e-4166-868b-ff94fa2010b9-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:37.460553 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:37.460524 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" Oct 30 13:04:37.460553 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:37.460538 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8" event={"ID":"6435f44e-980e-4166-868b-ff94fa2010b9","Type":"ContainerDied","Data":"03e900e2d373ae0cfaabe51fa4a08a6e55ccd4ccdeb334e6ae0b6bae4e6bad8c"} Oct 30 13:04:37.461102 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:37.460605 4413 scope.go:122] "RemoveContainer" containerID="e870bee964d94c7a593084b16f12abb8b855bd03eb8cb5f101b850720897f9f9" Oct 30 13:04:37.574309 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:37.574276 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8"] Oct 30 13:04:37.613719 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:37.613693 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0bda67-openshift-default-bdbd7b8d9-fr7n8"] Oct 30 13:04:37.658907 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:37.658869 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:37.973743 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:37.973708 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:38.750580 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:38.750555 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:38.868885 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:38.868856 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:39.021388 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:39.021314 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6435f44e-980e-4166-868b-ff94fa2010b9" path="/var/lib/kubelet/pods/6435f44e-980e-4166-868b-ff94fa2010b9/volumes" Oct 30 13:04:39.873479 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:39.873432 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:41.395278 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:41.395233 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:41.569714 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:41.569686 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:41.789509 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:41.789425 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:41.920426 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:41.920393 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:42.440016 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.439946 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" podUID="f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2" containerName="main" containerID="cri-o://5a853a069e585be2039228aee95ffbc34ac8762e12d25ca0517b32a1bb0530ff" gracePeriod=45 Oct 30 13:04:42.754790 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.754760 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 13:04:42.845720 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.845689 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n"] Oct 30 13:04:42.848981 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.848957 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-home\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-home\") pod \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " Oct 30 13:04:42.849132 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.849018 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-model-cache\") pod \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " Oct 30 13:04:42.849132 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.849092 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-tls-certs\" (UniqueName: \"kubernetes.io/secret/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-tls-certs\") pod \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " Oct 30 13:04:42.849132 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.849117 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-kube-api-access-hldx7\" (UniqueName: \"kubernetes.io/projected/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-kube-api-access-hldx7\") pod \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " Oct 30 13:04:42.849348 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.849159 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-dshm\") pod \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " Oct 30 13:04:42.849348 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.849229 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-tmp-dir\") pod \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\" (UID: \"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2\") " Oct 30 13:04:42.849348 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.849295 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-home" pod "f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2" (UID: "f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:42.849535 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.849367 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-model-cache" pod "f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2" (UID: "f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:42.849650 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.849620 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-tmp-dir" pod "f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2" (UID: "f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:42.849831 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.849817 4413 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:42.849871 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.849833 4413 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:42.849871 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.849843 4413 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:42.851679 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.851658 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-dshm" pod "f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2" (UID: "f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:04:42.852090 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.852071 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-kube-api-access-hldx7" pod "f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2" (UID: "f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2"). InnerVolumeSpecName "kube-api-access-hldx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:42.852148 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.852131 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-tls-certs" pod "f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2" (UID: "f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:04:42.951752 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.951708 4413 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:42.951752 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.951744 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hldx7\" (UniqueName: \"kubernetes.io/projected/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-kube-api-access-hldx7\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:42.951752 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.951758 4413 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:42.982011 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:42.981947 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:43.103493 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:43.103464 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:43.507456 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:43.507423 4413 generic.go:356] "Generic (PLEG): container finished" podID="f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2" containerID="5a853a069e585be2039228aee95ffbc34ac8762e12d25ca0517b32a1bb0530ff" exitCode=0 Oct 30 13:04:43.507965 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:43.507470 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" Oct 30 13:04:43.507965 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:43.507470 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" event={"ID":"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2","Type":"ContainerDied","Data":"5a853a069e585be2039228aee95ffbc34ac8762e12d25ca0517b32a1bb0530ff"} Oct 30 13:04:43.507965 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:43.507517 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct" event={"ID":"f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2","Type":"ContainerDied","Data":"ef4ce3394afc15a1965f2c2f8ecee83ef502285b2945bfdca5ec51389905c7b3"} Oct 30 13:04:43.507965 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:43.507543 4413 scope.go:122] "RemoveContainer" containerID="5a853a069e585be2039228aee95ffbc34ac8762e12d25ca0517b32a1bb0530ff" Oct 30 13:04:43.518177 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:43.518149 4413 scope.go:122] "RemoveContainer" containerID="5a853a069e585be2039228aee95ffbc34ac8762e12d25ca0517b32a1bb0530ff" Oct 30 13:04:43.518448 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:04:43.518431 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a853a069e585be2039228aee95ffbc34ac8762e12d25ca0517b32a1bb0530ff\": container with ID starting with 5a853a069e585be2039228aee95ffbc34ac8762e12d25ca0517b32a1bb0530ff not found: ID does not exist" containerID="5a853a069e585be2039228aee95ffbc34ac8762e12d25ca0517b32a1bb0530ff" Oct 30 13:04:43.518521 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:43.518455 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a853a069e585be2039228aee95ffbc34ac8762e12d25ca0517b32a1bb0530ff"} err="failed to get container status \"5a853a069e585be2039228aee95ffbc34ac8762e12d25ca0517b32a1bb0530ff\": rpc error: code = NotFound desc = could not find container \"5a853a069e585be2039228aee95ffbc34ac8762e12d25ca0517b32a1bb0530ff\": container with ID starting with 5a853a069e585be2039228aee95ffbc34ac8762e12d25ca0517b32a1bb0530ff not found: ID does not exist" Oct 30 13:04:43.560046 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:43.560017 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct"] Oct 30 13:04:43.566959 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:43.566932 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-befaa6/test-model-befaa6-kserve-855d4f4c89-qhpct"] Oct 30 13:04:43.632020 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:43.631985 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:45.024835 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:45.024761 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2" path="/var/lib/kubelet/pods/f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2/volumes" Oct 30 13:04:45.050381 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:45.050356 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:45.555211 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:45.555181 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:45.673978 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:45.673948 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:45.786270 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:45.786228 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:46.168559 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:46.168525 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:47.778737 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:47.778709 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:47.853643 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:47.853596 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n" podUID="1dde17ed-ff85-43dd-8cd7-a702a97bbd3b" containerName="payload-pre-processing" containerID="cri-o://6f62843a107106796831392e6e14b7916d292b3b2c988e1629c0aeee86e72854" gracePeriod=25 Oct 30 13:04:48.140987 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:48.140962 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n" Oct 30 13:04:48.280391 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:48.280358 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dde17ed-ff85-43dd-8cd7-a702a97bbd3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dde17ed-ff85-43dd-8cd7-a702a97bbd3b-plugins-config-volume\") pod \"1dde17ed-ff85-43dd-8cd7-a702a97bbd3b\" (UID: \"1dde17ed-ff85-43dd-8cd7-a702a97bbd3b\") " Oct 30 13:04:48.280391 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:48.280396 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dde17ed-ff85-43dd-8cd7-a702a97bbd3b-kube-api-access-9wsgj\" (UniqueName: \"kubernetes.io/projected/1dde17ed-ff85-43dd-8cd7-a702a97bbd3b-kube-api-access-9wsgj\") pod \"1dde17ed-ff85-43dd-8cd7-a702a97bbd3b\" (UID: \"1dde17ed-ff85-43dd-8cd7-a702a97bbd3b\") " Oct 30 13:04:48.280822 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:48.280793 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dde17ed-ff85-43dd-8cd7-a702a97bbd3b-plugins-config-volume" pod "1dde17ed-ff85-43dd-8cd7-a702a97bbd3b" (UID: "1dde17ed-ff85-43dd-8cd7-a702a97bbd3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:04:48.281073 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:48.281054 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dde17ed-ff85-43dd-8cd7-a702a97bbd3b-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:48.282805 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:48.282779 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dde17ed-ff85-43dd-8cd7-a702a97bbd3b-kube-api-access-9wsgj" pod "1dde17ed-ff85-43dd-8cd7-a702a97bbd3b" (UID: "1dde17ed-ff85-43dd-8cd7-a702a97bbd3b"). InnerVolumeSpecName "kube-api-access-9wsgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:04:48.383152 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:48.383117 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wsgj\" (UniqueName: \"kubernetes.io/projected/1dde17ed-ff85-43dd-8cd7-a702a97bbd3b-kube-api-access-9wsgj\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:04:48.410842 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:48.410797 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:48.545479 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:48.545400 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:48.547129 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:48.547105 4413 generic.go:356] "Generic (PLEG): container finished" podID="1dde17ed-ff85-43dd-8cd7-a702a97bbd3b" containerID="6f62843a107106796831392e6e14b7916d292b3b2c988e1629c0aeee86e72854" exitCode=0 Oct 30 13:04:48.547283 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:48.547168 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n" Oct 30 13:04:48.547283 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:48.547200 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n" event={"ID":"1dde17ed-ff85-43dd-8cd7-a702a97bbd3b","Type":"ContainerDied","Data":"6f62843a107106796831392e6e14b7916d292b3b2c988e1629c0aeee86e72854"} Oct 30 13:04:48.547377 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:48.547304 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n" event={"ID":"1dde17ed-ff85-43dd-8cd7-a702a97bbd3b","Type":"ContainerDied","Data":"9332a3e5e2e26d17289521f31e1100a079af39a4afad6171682453848ee9acac"} Oct 30 13:04:48.547377 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:48.547341 4413 scope.go:122] "RemoveContainer" containerID="6f62843a107106796831392e6e14b7916d292b3b2c988e1629c0aeee86e72854" Oct 30 13:04:48.558554 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:48.558536 4413 scope.go:122] "RemoveContainer" containerID="6f62843a107106796831392e6e14b7916d292b3b2c988e1629c0aeee86e72854" Oct 30 13:04:48.558834 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:04:48.558816 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f62843a107106796831392e6e14b7916d292b3b2c988e1629c0aeee86e72854\": container with ID starting with 6f62843a107106796831392e6e14b7916d292b3b2c988e1629c0aeee86e72854 not found: ID does not exist" containerID="6f62843a107106796831392e6e14b7916d292b3b2c988e1629c0aeee86e72854" Oct 30 13:04:48.558907 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:48.558846 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f62843a107106796831392e6e14b7916d292b3b2c988e1629c0aeee86e72854"} err="failed to get container status \"6f62843a107106796831392e6e14b7916d292b3b2c988e1629c0aeee86e72854\": rpc error: code = NotFound desc = could not find container \"6f62843a107106796831392e6e14b7916d292b3b2c988e1629c0aeee86e72854\": container with ID starting with 6f62843a107106796831392e6e14b7916d292b3b2c988e1629c0aeee86e72854 not found: ID does not exist" Oct 30 13:04:48.605183 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:48.605151 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n"] Oct 30 13:04:48.610068 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:48.610039 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-dbe123-6bf45ccd5f-8sf4n"] Oct 30 13:04:49.024964 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:49.024932 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dde17ed-ff85-43dd-8cd7-a702a97bbd3b" path="/var/lib/kubelet/pods/1dde17ed-ff85-43dd-8cd7-a702a97bbd3b/volumes" Oct 30 13:04:49.546424 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:49.546395 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:51.166119 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:51.166094 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:52.831432 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:52.831398 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:52.958524 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:52.958495 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:53.107917 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:53.107817 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:53.265242 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:53.265209 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:53.671656 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:53.671626 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:54.574228 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:54.574198 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:56.281812 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:56.281777 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:57.929272 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:57.929223 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:58.727149 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:58.727114 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj"] Oct 30 13:04:58.746346 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:58.746309 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll"] Oct 30 13:04:58.915440 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:58.915409 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:59.402976 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:59.402942 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:04:59.929869 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:04:59.929841 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:00.053512 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:00.053483 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:00.167194 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:00.167161 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:00.510695 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:00.510665 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:01.965183 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:01.965148 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:03.054229 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:03.054193 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:03.455087 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:03.455058 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:03.568906 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:03.568859 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:04.504160 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:04.504124 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:04.619139 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:04.619101 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:05.283612 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:05.283583 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:05.816307 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:05:05.816259 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:05:06.967136 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:06.967094 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:07.291361 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:07.291268 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:07.423449 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:07.423414 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:07.663988 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:07.663950 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:08.646678 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:08.646624 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:10.079866 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.079836 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj"] Oct 30 13:05:10.080310 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.080079 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" podUID="ea820a52-0cbd-4fc8-9539-4aa9810e1836" containerName="maas-api" containerID="cri-o://19005136cebf590cbb4fc84b1e0f28c200a5252f9977e334a80c0d630e737fb9" gracePeriod=30 Oct 30 13:05:10.201771 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.201718 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:10.376627 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.376589 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" Oct 30 13:05:10.436703 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.436655 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea820a52-0cbd-4fc8-9539-4aa9810e1836-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea820a52-0cbd-4fc8-9539-4aa9810e1836-openshift-service-ca\") pod \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\" (UID: \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\") " Oct 30 13:05:10.436903 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.436847 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea820a52-0cbd-4fc8-9539-4aa9810e1836-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea820a52-0cbd-4fc8-9539-4aa9810e1836-maas-api-tls\") pod \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\" (UID: \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\") " Oct 30 13:05:10.437033 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.437013 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea820a52-0cbd-4fc8-9539-4aa9810e1836-kube-api-access-n7wj9\" (UniqueName: \"kubernetes.io/projected/ea820a52-0cbd-4fc8-9539-4aa9810e1836-kube-api-access-n7wj9\") pod \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\" (UID: \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\") " Oct 30 13:05:10.437286 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.437172 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea820a52-0cbd-4fc8-9539-4aa9810e1836-openshift-service-ca" pod "ea820a52-0cbd-4fc8-9539-4aa9810e1836" (UID: "ea820a52-0cbd-4fc8-9539-4aa9810e1836"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:10.437368 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.437327 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea820a52-0cbd-4fc8-9539-4aa9810e1836-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea820a52-0cbd-4fc8-9539-4aa9810e1836-metrics-certs\") pod \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\" (UID: \"ea820a52-0cbd-4fc8-9539-4aa9810e1836\") " Oct 30 13:05:10.438326 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.438300 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea820a52-0cbd-4fc8-9539-4aa9810e1836-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:10.439982 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.439951 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea820a52-0cbd-4fc8-9539-4aa9810e1836-metrics-certs" pod "ea820a52-0cbd-4fc8-9539-4aa9810e1836" (UID: "ea820a52-0cbd-4fc8-9539-4aa9810e1836"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:10.440108 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.440002 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea820a52-0cbd-4fc8-9539-4aa9810e1836-maas-api-tls" pod "ea820a52-0cbd-4fc8-9539-4aa9810e1836" (UID: "ea820a52-0cbd-4fc8-9539-4aa9810e1836"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:10.440108 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.440048 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea820a52-0cbd-4fc8-9539-4aa9810e1836-kube-api-access-n7wj9" pod "ea820a52-0cbd-4fc8-9539-4aa9810e1836" (UID: "ea820a52-0cbd-4fc8-9539-4aa9810e1836"). InnerVolumeSpecName "kube-api-access-n7wj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:10.539884 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.539846 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7wj9\" (UniqueName: \"kubernetes.io/projected/ea820a52-0cbd-4fc8-9539-4aa9810e1836-kube-api-access-n7wj9\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:10.539884 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.539881 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea820a52-0cbd-4fc8-9539-4aa9810e1836-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:10.539884 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.539895 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea820a52-0cbd-4fc8-9539-4aa9810e1836-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:10.716311 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.716276 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:10.719339 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.719313 4413 generic.go:356] "Generic (PLEG): container finished" podID="ea820a52-0cbd-4fc8-9539-4aa9810e1836" containerID="19005136cebf590cbb4fc84b1e0f28c200a5252f9977e334a80c0d630e737fb9" exitCode=0 Oct 30 13:05:10.719491 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.719369 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" Oct 30 13:05:10.719491 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.719392 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" event={"ID":"ea820a52-0cbd-4fc8-9539-4aa9810e1836","Type":"ContainerDied","Data":"19005136cebf590cbb4fc84b1e0f28c200a5252f9977e334a80c0d630e737fb9"} Oct 30 13:05:10.719491 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.719455 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj" event={"ID":"ea820a52-0cbd-4fc8-9539-4aa9810e1836","Type":"ContainerDied","Data":"b8e1b7bda5ea68a8363afc3f7165603fe50d9a5d22b7dcc5619a17b1c4d2b9ae"} Oct 30 13:05:10.719491 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.719487 4413 scope.go:122] "RemoveContainer" containerID="19005136cebf590cbb4fc84b1e0f28c200a5252f9977e334a80c0d630e737fb9" Oct 30 13:05:10.731812 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.731782 4413 scope.go:122] "RemoveContainer" containerID="19005136cebf590cbb4fc84b1e0f28c200a5252f9977e334a80c0d630e737fb9" Oct 30 13:05:10.732135 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:05:10.732115 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19005136cebf590cbb4fc84b1e0f28c200a5252f9977e334a80c0d630e737fb9\": container with ID starting with 19005136cebf590cbb4fc84b1e0f28c200a5252f9977e334a80c0d630e737fb9 not found: ID does not exist" containerID="19005136cebf590cbb4fc84b1e0f28c200a5252f9977e334a80c0d630e737fb9" Oct 30 13:05:10.732197 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.732144 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19005136cebf590cbb4fc84b1e0f28c200a5252f9977e334a80c0d630e737fb9"} err="failed to get container status \"19005136cebf590cbb4fc84b1e0f28c200a5252f9977e334a80c0d630e737fb9\": rpc error: code = NotFound desc = could not find container \"19005136cebf590cbb4fc84b1e0f28c200a5252f9977e334a80c0d630e737fb9\": container with ID starting with 19005136cebf590cbb4fc84b1e0f28c200a5252f9977e334a80c0d630e737fb9 not found: ID does not exist" Oct 30 13:05:10.774163 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.774082 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj"] Oct 30 13:05:10.785711 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.785671 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b67f8-fc49f5456-fhwtj"] Oct 30 13:05:10.840455 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.840420 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:10.974763 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:10.974651 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:11.022851 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:11.022812 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea820a52-0cbd-4fc8-9539-4aa9810e1836" path="/var/lib/kubelet/pods/ea820a52-0cbd-4fc8-9539-4aa9810e1836/volumes" Oct 30 13:05:11.324520 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:11.324380 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:12.618623 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:12.618590 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:13.750030 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:13.749986 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" containerID="cri-o://7f3652db80dc696daff68775b47bb2f668c9165b6058ba30530bd8a876779441" gracePeriod=45 Oct 30 13:05:13.768913 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:13.768863 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerName="main" containerID="cri-o://8759467d1ce0f0147f92eaab43eb2ad310784f198244284ba559cab014dc76be" gracePeriod=45 Oct 30 13:05:14.073224 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.073194 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 13:05:14.077331 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.077303 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 13:05:14.195404 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.195363 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-tls-certs\") pod \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " Oct 30 13:05:14.195632 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.195447 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-home\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-home\") pod \"cdebed52-2014-447e-975a-3b6c02c13535\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " Oct 30 13:05:14.195632 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.195549 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-kube-api-access-9lf9k\" (UniqueName: \"kubernetes.io/projected/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-kube-api-access-9lf9k\") pod \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " Oct 30 13:05:14.195632 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.195590 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdebed52-2014-447e-975a-3b6c02c13535-tls-certs\" (UniqueName: \"kubernetes.io/secret/cdebed52-2014-447e-975a-3b6c02c13535-tls-certs\") pod \"cdebed52-2014-447e-975a-3b6c02c13535\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " Oct 30 13:05:14.195835 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.195673 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-home\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-home\") pod \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " Oct 30 13:05:14.195835 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.195710 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-model-cache\") pod \"cdebed52-2014-447e-975a-3b6c02c13535\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " Oct 30 13:05:14.195835 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.195770 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdebed52-2014-447e-975a-3b6c02c13535-kube-api-access-vpctg\" (UniqueName: \"kubernetes.io/projected/cdebed52-2014-447e-975a-3b6c02c13535-kube-api-access-vpctg\") pod \"cdebed52-2014-447e-975a-3b6c02c13535\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " Oct 30 13:05:14.195835 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.195815 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-model-cache\") pod \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " Oct 30 13:05:14.196065 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.195889 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-dshm\") pod \"cdebed52-2014-447e-975a-3b6c02c13535\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " Oct 30 13:05:14.196241 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.196212 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-model-cache" pod "cdebed52-2014-447e-975a-3b6c02c13535" (UID: "cdebed52-2014-447e-975a-3b6c02c13535"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:14.196427 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.196401 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-home" pod "85c3372a-f4c6-450f-8de9-5a826d3ee1f6" (UID: "85c3372a-f4c6-450f-8de9-5a826d3ee1f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:14.196683 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.196470 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-model-cache" pod "85c3372a-f4c6-450f-8de9-5a826d3ee1f6" (UID: "85c3372a-f4c6-450f-8de9-5a826d3ee1f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:14.196683 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.196584 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-home" pod "cdebed52-2014-447e-975a-3b6c02c13535" (UID: "cdebed52-2014-447e-975a-3b6c02c13535"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:14.197040 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.197019 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-dshm\") pod \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " Oct 30 13:05:14.197188 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.197171 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-tmp-dir\") pod \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\" (UID: \"85c3372a-f4c6-450f-8de9-5a826d3ee1f6\") " Oct 30 13:05:14.197427 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.197395 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-tmp-dir\") pod \"cdebed52-2014-447e-975a-3b6c02c13535\" (UID: \"cdebed52-2014-447e-975a-3b6c02c13535\") " Oct 30 13:05:14.197886 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.197856 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-tmp-dir" pod "85c3372a-f4c6-450f-8de9-5a826d3ee1f6" (UID: "85c3372a-f4c6-450f-8de9-5a826d3ee1f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:14.198432 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.198410 4413 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.198557 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.198545 4413 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.198647 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.198637 4413 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.198747 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.198735 4413 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.198889 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.198878 4413 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.198996 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.198656 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-tmp-dir" pod "cdebed52-2014-447e-975a-3b6c02c13535" (UID: "cdebed52-2014-447e-975a-3b6c02c13535"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:14.200300 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.200264 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdebed52-2014-447e-975a-3b6c02c13535-tls-certs" pod "cdebed52-2014-447e-975a-3b6c02c13535" (UID: "cdebed52-2014-447e-975a-3b6c02c13535"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:14.201151 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.201114 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-tls-certs" pod "85c3372a-f4c6-450f-8de9-5a826d3ee1f6" (UID: "85c3372a-f4c6-450f-8de9-5a826d3ee1f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:14.201430 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.201403 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-dshm" pod "cdebed52-2014-447e-975a-3b6c02c13535" (UID: "cdebed52-2014-447e-975a-3b6c02c13535"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:14.201596 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.201566 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-kube-api-access-9lf9k" pod "85c3372a-f4c6-450f-8de9-5a826d3ee1f6" (UID: "85c3372a-f4c6-450f-8de9-5a826d3ee1f6"). InnerVolumeSpecName "kube-api-access-9lf9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:14.201874 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.201841 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdebed52-2014-447e-975a-3b6c02c13535-kube-api-access-vpctg" pod "cdebed52-2014-447e-975a-3b6c02c13535" (UID: "cdebed52-2014-447e-975a-3b6c02c13535"). InnerVolumeSpecName "kube-api-access-vpctg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:14.203719 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.203690 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-dshm" pod "85c3372a-f4c6-450f-8de9-5a826d3ee1f6" (UID: "85c3372a-f4c6-450f-8de9-5a826d3ee1f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:14.300632 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.300533 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lf9k\" (UniqueName: \"kubernetes.io/projected/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-kube-api-access-9lf9k\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.300632 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.300569 4413 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdebed52-2014-447e-975a-3b6c02c13535-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.300632 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.300591 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpctg\" (UniqueName: \"kubernetes.io/projected/cdebed52-2014-447e-975a-3b6c02c13535-kube-api-access-vpctg\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.300632 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.300602 4413 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.300632 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.300613 4413 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.300632 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.300624 4413 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdebed52-2014-447e-975a-3b6c02c13535-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.300632 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.300634 4413 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85c3372a-f4c6-450f-8de9-5a826d3ee1f6-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:14.372578 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.372538 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:14.752458 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.752425 4413 generic.go:356] "Generic (PLEG): container finished" podID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerID="8759467d1ce0f0147f92eaab43eb2ad310784f198244284ba559cab014dc76be" exitCode=0 Oct 30 13:05:14.752990 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.752478 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" Oct 30 13:05:14.752990 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.752514 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" event={"ID":"85c3372a-f4c6-450f-8de9-5a826d3ee1f6","Type":"ContainerDied","Data":"8759467d1ce0f0147f92eaab43eb2ad310784f198244284ba559cab014dc76be"} Oct 30 13:05:14.752990 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.752557 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll" event={"ID":"85c3372a-f4c6-450f-8de9-5a826d3ee1f6","Type":"ContainerDied","Data":"e96ca24dc6d23e6497f21eda85182f969ae1d4e1496c20e42ec3072a895e8922"} Oct 30 13:05:14.752990 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.752577 4413 scope.go:122] "RemoveContainer" containerID="8759467d1ce0f0147f92eaab43eb2ad310784f198244284ba559cab014dc76be" Oct 30 13:05:14.754159 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.754125 4413 generic.go:356] "Generic (PLEG): container finished" podID="cdebed52-2014-447e-975a-3b6c02c13535" containerID="7f3652db80dc696daff68775b47bb2f668c9165b6058ba30530bd8a876779441" exitCode=0 Oct 30 13:05:14.754239 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.754176 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" Oct 30 13:05:14.754239 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.754206 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" event={"ID":"cdebed52-2014-447e-975a-3b6c02c13535","Type":"ContainerDied","Data":"7f3652db80dc696daff68775b47bb2f668c9165b6058ba30530bd8a876779441"} Oct 30 13:05:14.754373 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.754280 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj" event={"ID":"cdebed52-2014-447e-975a-3b6c02c13535","Type":"ContainerDied","Data":"ff4d87c4affbc7230efc61efa90963d0133e6f6244663ee35973319b0c122cf0"} Oct 30 13:05:14.763908 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.763883 4413 scope.go:122] "RemoveContainer" containerID="8759467d1ce0f0147f92eaab43eb2ad310784f198244284ba559cab014dc76be" Oct 30 13:05:14.764336 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:05:14.764314 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8759467d1ce0f0147f92eaab43eb2ad310784f198244284ba559cab014dc76be\": container with ID starting with 8759467d1ce0f0147f92eaab43eb2ad310784f198244284ba559cab014dc76be not found: ID does not exist" containerID="8759467d1ce0f0147f92eaab43eb2ad310784f198244284ba559cab014dc76be" Oct 30 13:05:14.764430 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.764349 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8759467d1ce0f0147f92eaab43eb2ad310784f198244284ba559cab014dc76be"} err="failed to get container status \"8759467d1ce0f0147f92eaab43eb2ad310784f198244284ba559cab014dc76be\": rpc error: code = NotFound desc = could not find container \"8759467d1ce0f0147f92eaab43eb2ad310784f198244284ba559cab014dc76be\": container with ID starting with 8759467d1ce0f0147f92eaab43eb2ad310784f198244284ba559cab014dc76be not found: ID does not exist" Oct 30 13:05:14.764430 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.764374 4413 scope.go:122] "RemoveContainer" containerID="7f3652db80dc696daff68775b47bb2f668c9165b6058ba30530bd8a876779441" Oct 30 13:05:14.780939 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.780912 4413 scope.go:122] "RemoveContainer" containerID="7f3652db80dc696daff68775b47bb2f668c9165b6058ba30530bd8a876779441" Oct 30 13:05:14.781332 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:05:14.781305 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f3652db80dc696daff68775b47bb2f668c9165b6058ba30530bd8a876779441\": container with ID starting with 7f3652db80dc696daff68775b47bb2f668c9165b6058ba30530bd8a876779441 not found: ID does not exist" containerID="7f3652db80dc696daff68775b47bb2f668c9165b6058ba30530bd8a876779441" Oct 30 13:05:14.781453 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.781342 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3652db80dc696daff68775b47bb2f668c9165b6058ba30530bd8a876779441"} err="failed to get container status \"7f3652db80dc696daff68775b47bb2f668c9165b6058ba30530bd8a876779441\": rpc error: code = NotFound desc = could not find container \"7f3652db80dc696daff68775b47bb2f668c9165b6058ba30530bd8a876779441\": container with ID starting with 7f3652db80dc696daff68775b47bb2f668c9165b6058ba30530bd8a876779441 not found: ID does not exist" Oct 30 13:05:14.819555 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.819518 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj"] Oct 30 13:05:14.830795 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.830768 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/distinct-w1-c1b1e8-kserve-bf44dcdfb-gnzkj"] Oct 30 13:05:14.833453 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.833422 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:14.856576 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.856543 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll"] Oct 30 13:05:14.860182 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:14.860154 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c1b1e8/simulator-w1-c1b1e8-kserve-75ccfbcb-lj6ll"] Oct 30 13:05:15.021827 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:15.021748 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" path="/var/lib/kubelet/pods/85c3372a-f4c6-450f-8de9-5a826d3ee1f6/volumes" Oct 30 13:05:15.022261 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:15.022237 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdebed52-2014-447e-975a-3b6c02c13535" path="/var/lib/kubelet/pods/cdebed52-2014-447e-975a-3b6c02c13535/volumes" Oct 30 13:05:16.069136 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:16.069069 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:17.926618 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:17.926582 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:18.544898 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:18.544867 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:18.803747 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:18.803639 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:18.817425 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:18.817386 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj"] Oct 30 13:05:18.978731 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:18.978649 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:19.112609 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:19.112512 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:19.770994 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:19.770948 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:19.955150 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:19.955114 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:20.077464 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:20.077387 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:21.577525 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:21.577479 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:23.124453 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:23.124419 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf"] Oct 30 13:05:23.229537 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:23.229453 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:23.683420 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:23.683383 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:23.818541 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:23.818500 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" podUID="935f35f7-28c0-4d05-ab8e-1a2042d7aab4" containerName="payload-pre-processing" containerID="cri-o://ece04963b3a4e083f12f1f5980b032c461b1e3470c7a472948a3c4f1165247b6" gracePeriod=25 Oct 30 13:05:25.348644 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:25.348612 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:27.162131 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:27.162104 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:28.885828 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:28.885797 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:29.997497 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:29.997464 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm"] Oct 30 13:05:30.135547 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.135493 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:30.251623 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.251531 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:30.252283 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.252239 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv"] Oct 30 13:05:30.252541 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.252512 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" podUID="4c30ec11-6046-4da2-944a-c45f40848016" containerName="maas-api" containerID="cri-o://4ed9a101ac6a1e51189410919d185a34cf3ef3f1a226d06c7e39bc0ce7de39e2" gracePeriod=30 Oct 30 13:05:30.393485 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.393446 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:30.530128 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.530100 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" Oct 30 13:05:30.622668 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.622636 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c30ec11-6046-4da2-944a-c45f40848016-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c30ec11-6046-4da2-944a-c45f40848016-openshift-service-ca\") pod \"4c30ec11-6046-4da2-944a-c45f40848016\" (UID: \"4c30ec11-6046-4da2-944a-c45f40848016\") " Oct 30 13:05:30.622820 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.622740 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c30ec11-6046-4da2-944a-c45f40848016-kube-api-access-fdh6k\" (UniqueName: \"kubernetes.io/projected/4c30ec11-6046-4da2-944a-c45f40848016-kube-api-access-fdh6k\") pod \"4c30ec11-6046-4da2-944a-c45f40848016\" (UID: \"4c30ec11-6046-4da2-944a-c45f40848016\") " Oct 30 13:05:30.622820 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.622809 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c30ec11-6046-4da2-944a-c45f40848016-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c30ec11-6046-4da2-944a-c45f40848016-metrics-certs\") pod \"4c30ec11-6046-4da2-944a-c45f40848016\" (UID: \"4c30ec11-6046-4da2-944a-c45f40848016\") " Oct 30 13:05:30.622920 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.622885 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c30ec11-6046-4da2-944a-c45f40848016-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c30ec11-6046-4da2-944a-c45f40848016-maas-api-tls\") pod \"4c30ec11-6046-4da2-944a-c45f40848016\" (UID: \"4c30ec11-6046-4da2-944a-c45f40848016\") " Oct 30 13:05:30.623069 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.623048 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c30ec11-6046-4da2-944a-c45f40848016-openshift-service-ca" pod "4c30ec11-6046-4da2-944a-c45f40848016" (UID: "4c30ec11-6046-4da2-944a-c45f40848016"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:30.623416 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.623393 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c30ec11-6046-4da2-944a-c45f40848016-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:30.625071 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.625039 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c30ec11-6046-4da2-944a-c45f40848016-kube-api-access-fdh6k" pod "4c30ec11-6046-4da2-944a-c45f40848016" (UID: "4c30ec11-6046-4da2-944a-c45f40848016"). InnerVolumeSpecName "kube-api-access-fdh6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:30.625160 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.625044 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c30ec11-6046-4da2-944a-c45f40848016-maas-api-tls" pod "4c30ec11-6046-4da2-944a-c45f40848016" (UID: "4c30ec11-6046-4da2-944a-c45f40848016"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:30.625160 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.625079 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c30ec11-6046-4da2-944a-c45f40848016-metrics-certs" pod "4c30ec11-6046-4da2-944a-c45f40848016" (UID: "4c30ec11-6046-4da2-944a-c45f40848016"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:30.725237 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.725156 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdh6k\" (UniqueName: \"kubernetes.io/projected/4c30ec11-6046-4da2-944a-c45f40848016-kube-api-access-fdh6k\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:30.725237 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.725195 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c30ec11-6046-4da2-944a-c45f40848016-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:30.725237 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.725210 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c30ec11-6046-4da2-944a-c45f40848016-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:30.873607 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.873574 4413 generic.go:356] "Generic (PLEG): container finished" podID="4c30ec11-6046-4da2-944a-c45f40848016" containerID="4ed9a101ac6a1e51189410919d185a34cf3ef3f1a226d06c7e39bc0ce7de39e2" exitCode=0 Oct 30 13:05:30.873782 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.873625 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" Oct 30 13:05:30.873782 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.873666 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" event={"ID":"4c30ec11-6046-4da2-944a-c45f40848016","Type":"ContainerDied","Data":"4ed9a101ac6a1e51189410919d185a34cf3ef3f1a226d06c7e39bc0ce7de39e2"} Oct 30 13:05:30.873782 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.873733 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv" event={"ID":"4c30ec11-6046-4da2-944a-c45f40848016","Type":"ContainerDied","Data":"4e592f9dbb339bb7161dd7382e852e27ac8e0c161297278594a291cc04a8bba4"} Oct 30 13:05:30.873782 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.873762 4413 scope.go:122] "RemoveContainer" containerID="4ed9a101ac6a1e51189410919d185a34cf3ef3f1a226d06c7e39bc0ce7de39e2" Oct 30 13:05:30.885140 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.885120 4413 scope.go:122] "RemoveContainer" containerID="4ed9a101ac6a1e51189410919d185a34cf3ef3f1a226d06c7e39bc0ce7de39e2" Oct 30 13:05:30.885483 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:05:30.885462 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ed9a101ac6a1e51189410919d185a34cf3ef3f1a226d06c7e39bc0ce7de39e2\": container with ID starting with 4ed9a101ac6a1e51189410919d185a34cf3ef3f1a226d06c7e39bc0ce7de39e2 not found: ID does not exist" containerID="4ed9a101ac6a1e51189410919d185a34cf3ef3f1a226d06c7e39bc0ce7de39e2" Oct 30 13:05:30.885565 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.885489 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed9a101ac6a1e51189410919d185a34cf3ef3f1a226d06c7e39bc0ce7de39e2"} err="failed to get container status \"4ed9a101ac6a1e51189410919d185a34cf3ef3f1a226d06c7e39bc0ce7de39e2\": rpc error: code = NotFound desc = could not find container \"4ed9a101ac6a1e51189410919d185a34cf3ef3f1a226d06c7e39bc0ce7de39e2\": container with ID starting with 4ed9a101ac6a1e51189410919d185a34cf3ef3f1a226d06c7e39bc0ce7de39e2 not found: ID does not exist" Oct 30 13:05:30.896026 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.896009 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:30.925979 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.925956 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv"] Oct 30 13:05:30.937678 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:30.937649 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0de4b9-67db4d7984-8nczv"] Oct 30 13:05:31.014651 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:31.014621 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:31.020615 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:31.020590 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c30ec11-6046-4da2-944a-c45f40848016" path="/var/lib/kubelet/pods/4c30ec11-6046-4da2-944a-c45f40848016/volumes" Oct 30 13:05:31.127101 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:31.127027 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:31.596292 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:31.596183 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:32.287896 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:32.287860 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" podUID="935f35f7-28c0-4d05-ab8e-1a2042d7aab4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:05:32.874600 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:32.874562 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:34.716972 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:34.716928 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:34.998086 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:34.997996 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm" podUID="a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39" containerName="payload-processing" containerID="cri-o://f01c115fbd197de2feb22b9585dff1c418b3ea5dba7b19dfeb6b10f0f9bab626" gracePeriod=25 Oct 30 13:05:35.272784 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:35.272760 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm" Oct 30 13:05:35.386603 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:35.386568 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39-kube-api-access-9blrn\" (UniqueName: \"kubernetes.io/projected/a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39-kube-api-access-9blrn\") pod \"a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39\" (UID: \"a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39\") " Oct 30 13:05:35.386822 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:35.386735 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39-plugins-config-volume\") pod \"a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39\" (UID: \"a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39\") " Oct 30 13:05:35.387244 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:35.387181 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39-plugins-config-volume" pod "a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39" (UID: "a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:35.387520 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:35.387407 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:35.388887 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:35.388865 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39-kube-api-access-9blrn" pod "a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39" (UID: "a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39"). InnerVolumeSpecName "kube-api-access-9blrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:35.488846 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:35.488819 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9blrn\" (UniqueName: \"kubernetes.io/projected/a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39-kube-api-access-9blrn\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:35.814571 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:05:35.814535 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:05:35.909569 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:35.909535 4413 generic.go:356] "Generic (PLEG): container finished" podID="a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39" containerID="f01c115fbd197de2feb22b9585dff1c418b3ea5dba7b19dfeb6b10f0f9bab626" exitCode=0 Oct 30 13:05:35.909723 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:35.909585 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm" Oct 30 13:05:35.909723 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:35.909616 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm" event={"ID":"a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39","Type":"ContainerDied","Data":"f01c115fbd197de2feb22b9585dff1c418b3ea5dba7b19dfeb6b10f0f9bab626"} Oct 30 13:05:35.909723 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:35.909676 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm" event={"ID":"a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39","Type":"ContainerDied","Data":"512aa26d735ceeaea9669d238de4bb5310f70013a2be0cfa520f369523c3241b"} Oct 30 13:05:35.909723 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:35.909698 4413 scope.go:122] "RemoveContainer" containerID="f01c115fbd197de2feb22b9585dff1c418b3ea5dba7b19dfeb6b10f0f9bab626" Oct 30 13:05:35.920173 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:35.920157 4413 scope.go:122] "RemoveContainer" containerID="f01c115fbd197de2feb22b9585dff1c418b3ea5dba7b19dfeb6b10f0f9bab626" Oct 30 13:05:35.920490 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:05:35.920469 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f01c115fbd197de2feb22b9585dff1c418b3ea5dba7b19dfeb6b10f0f9bab626\": container with ID starting with f01c115fbd197de2feb22b9585dff1c418b3ea5dba7b19dfeb6b10f0f9bab626 not found: ID does not exist" containerID="f01c115fbd197de2feb22b9585dff1c418b3ea5dba7b19dfeb6b10f0f9bab626" Oct 30 13:05:35.920547 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:35.920498 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01c115fbd197de2feb22b9585dff1c418b3ea5dba7b19dfeb6b10f0f9bab626"} err="failed to get container status \"f01c115fbd197de2feb22b9585dff1c418b3ea5dba7b19dfeb6b10f0f9bab626\": rpc error: code = NotFound desc = could not find container \"f01c115fbd197de2feb22b9585dff1c418b3ea5dba7b19dfeb6b10f0f9bab626\": container with ID starting with f01c115fbd197de2feb22b9585dff1c418b3ea5dba7b19dfeb6b10f0f9bab626 not found: ID does not exist" Oct 30 13:05:35.964132 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:35.964103 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm"] Oct 30 13:05:35.974155 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:35.974131 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-befaa6-56d77cf7d4-t65jm"] Oct 30 13:05:36.629301 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:36.629235 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:36.766162 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:36.766125 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:37.021869 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:37.021787 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39" path="/var/lib/kubelet/pods/a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39/volumes" Oct 30 13:05:37.326872 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:37.326746 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:38.152965 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.152902 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" podUID="a8038e2f-a57c-490f-9206-84b5c379830a" containerName="main" containerID="cri-o://a1c0522e85627175016c3ae12dd4029239ed56af4446b05cad742cc32e30e110" gracePeriod=45 Oct 30 13:05:38.438597 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.438564 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:38.440177 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.440158 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 13:05:38.543391 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.543362 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8038e2f-a57c-490f-9206-84b5c379830a-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8038e2f-a57c-490f-9206-84b5c379830a-tls-certs\") pod \"a8038e2f-a57c-490f-9206-84b5c379830a\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " Oct 30 13:05:38.543715 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.543432 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8038e2f-a57c-490f-9206-84b5c379830a-kube-api-access-mbf6x\" (UniqueName: \"kubernetes.io/projected/a8038e2f-a57c-490f-9206-84b5c379830a-kube-api-access-mbf6x\") pod \"a8038e2f-a57c-490f-9206-84b5c379830a\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " Oct 30 13:05:38.543715 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.543507 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-model-cache\") pod \"a8038e2f-a57c-490f-9206-84b5c379830a\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " Oct 30 13:05:38.543715 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.543612 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-dshm\") pod \"a8038e2f-a57c-490f-9206-84b5c379830a\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " Oct 30 13:05:38.543715 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.543683 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-home\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-home\") pod \"a8038e2f-a57c-490f-9206-84b5c379830a\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " Oct 30 13:05:38.543963 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.543740 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-model-cache" pod "a8038e2f-a57c-490f-9206-84b5c379830a" (UID: "a8038e2f-a57c-490f-9206-84b5c379830a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:38.543963 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.543824 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-tmp-dir\") pod \"a8038e2f-a57c-490f-9206-84b5c379830a\" (UID: \"a8038e2f-a57c-490f-9206-84b5c379830a\") " Oct 30 13:05:38.544073 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.543973 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-home" pod "a8038e2f-a57c-490f-9206-84b5c379830a" (UID: "a8038e2f-a57c-490f-9206-84b5c379830a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:38.544073 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.544037 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-tmp-dir" pod "a8038e2f-a57c-490f-9206-84b5c379830a" (UID: "a8038e2f-a57c-490f-9206-84b5c379830a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:38.544412 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.544396 4413 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:38.544412 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.544412 4413 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:38.544553 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.544421 4413 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:38.545939 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.545917 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8038e2f-a57c-490f-9206-84b5c379830a-tls-certs" pod "a8038e2f-a57c-490f-9206-84b5c379830a" (UID: "a8038e2f-a57c-490f-9206-84b5c379830a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:38.545939 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.545924 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-dshm" pod "a8038e2f-a57c-490f-9206-84b5c379830a" (UID: "a8038e2f-a57c-490f-9206-84b5c379830a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:05:38.546073 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.545956 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8038e2f-a57c-490f-9206-84b5c379830a-kube-api-access-mbf6x" pod "a8038e2f-a57c-490f-9206-84b5c379830a" (UID: "a8038e2f-a57c-490f-9206-84b5c379830a"). InnerVolumeSpecName "kube-api-access-mbf6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:38.645456 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.645409 4413 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8038e2f-a57c-490f-9206-84b5c379830a-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:38.645456 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.645445 4413 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8038e2f-a57c-490f-9206-84b5c379830a-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:38.645456 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.645462 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbf6x\" (UniqueName: \"kubernetes.io/projected/a8038e2f-a57c-490f-9206-84b5c379830a-kube-api-access-mbf6x\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:38.934392 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.934349 4413 generic.go:356] "Generic (PLEG): container finished" podID="a8038e2f-a57c-490f-9206-84b5c379830a" containerID="a1c0522e85627175016c3ae12dd4029239ed56af4446b05cad742cc32e30e110" exitCode=0 Oct 30 13:05:38.934568 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.934402 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" Oct 30 13:05:38.934568 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.934398 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" event={"ID":"a8038e2f-a57c-490f-9206-84b5c379830a","Type":"ContainerDied","Data":"a1c0522e85627175016c3ae12dd4029239ed56af4446b05cad742cc32e30e110"} Oct 30 13:05:38.934568 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.934532 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf" event={"ID":"a8038e2f-a57c-490f-9206-84b5c379830a","Type":"ContainerDied","Data":"d03b41e0da71f98cb53074fcece6ad189ecbac8e9ea8d4864a9a562310da4f11"} Oct 30 13:05:38.934568 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.934566 4413 scope.go:122] "RemoveContainer" containerID="a1c0522e85627175016c3ae12dd4029239ed56af4446b05cad742cc32e30e110" Oct 30 13:05:38.948687 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.948671 4413 scope.go:122] "RemoveContainer" containerID="a1c0522e85627175016c3ae12dd4029239ed56af4446b05cad742cc32e30e110" Oct 30 13:05:38.948974 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:05:38.948948 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c0522e85627175016c3ae12dd4029239ed56af4446b05cad742cc32e30e110\": container with ID starting with a1c0522e85627175016c3ae12dd4029239ed56af4446b05cad742cc32e30e110 not found: ID does not exist" containerID="a1c0522e85627175016c3ae12dd4029239ed56af4446b05cad742cc32e30e110" Oct 30 13:05:38.949054 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.948976 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c0522e85627175016c3ae12dd4029239ed56af4446b05cad742cc32e30e110"} err="failed to get container status \"a1c0522e85627175016c3ae12dd4029239ed56af4446b05cad742cc32e30e110\": rpc error: code = NotFound desc = could not find container \"a1c0522e85627175016c3ae12dd4029239ed56af4446b05cad742cc32e30e110\": container with ID starting with a1c0522e85627175016c3ae12dd4029239ed56af4446b05cad742cc32e30e110 not found: ID does not exist" Oct 30 13:05:38.987626 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.987597 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf"] Oct 30 13:05:38.999734 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:38.999702 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1333c8/test-model-1333c8-kserve-5858cd6c97-x88gf"] Oct 30 13:05:39.022632 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:39.022608 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8038e2f-a57c-490f-9206-84b5c379830a" path="/var/lib/kubelet/pods/a8038e2f-a57c-490f-9206-84b5c379830a/volumes" Oct 30 13:05:39.119932 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:39.119904 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:39.235199 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:39.235111 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:39.474450 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:39.474418 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:40.366563 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:40.366537 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:41.455919 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:41.455885 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:42.230945 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:42.230909 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:42.288758 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:42.288722 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" podUID="935f35f7-28c0-4d05-ab8e-1a2042d7aab4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:05:43.612264 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:43.612226 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:43.730879 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:43.730845 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:43.977045 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:43.977007 4413 generic.go:356] "Generic (PLEG): container finished" podID="935f35f7-28c0-4d05-ab8e-1a2042d7aab4" containerID="ece04963b3a4e083f12f1f5980b032c461b1e3470c7a472948a3c4f1165247b6" exitCode=0 Oct 30 13:05:43.977236 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:43.977096 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" event={"ID":"935f35f7-28c0-4d05-ab8e-1a2042d7aab4","Type":"ContainerDied","Data":"ece04963b3a4e083f12f1f5980b032c461b1e3470c7a472948a3c4f1165247b6"} Oct 30 13:05:43.983549 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:43.983460 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:44.111541 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:44.111519 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" Oct 30 13:05:44.217681 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:44.217594 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/935f35f7-28c0-4d05-ab8e-1a2042d7aab4-kube-api-access-b278n\" (UniqueName: \"kubernetes.io/projected/935f35f7-28c0-4d05-ab8e-1a2042d7aab4-kube-api-access-b278n\") pod \"935f35f7-28c0-4d05-ab8e-1a2042d7aab4\" (UID: \"935f35f7-28c0-4d05-ab8e-1a2042d7aab4\") " Oct 30 13:05:44.217926 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:44.217910 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935f35f7-28c0-4d05-ab8e-1a2042d7aab4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935f35f7-28c0-4d05-ab8e-1a2042d7aab4-plugins-config-volume\") pod \"935f35f7-28c0-4d05-ab8e-1a2042d7aab4\" (UID: \"935f35f7-28c0-4d05-ab8e-1a2042d7aab4\") " Oct 30 13:05:44.218286 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:44.218244 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935f35f7-28c0-4d05-ab8e-1a2042d7aab4-plugins-config-volume" pod "935f35f7-28c0-4d05-ab8e-1a2042d7aab4" (UID: "935f35f7-28c0-4d05-ab8e-1a2042d7aab4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:44.218419 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:44.218406 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935f35f7-28c0-4d05-ab8e-1a2042d7aab4-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:44.220533 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:44.220503 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935f35f7-28c0-4d05-ab8e-1a2042d7aab4-kube-api-access-b278n" pod "935f35f7-28c0-4d05-ab8e-1a2042d7aab4" (UID: "935f35f7-28c0-4d05-ab8e-1a2042d7aab4"). InnerVolumeSpecName "kube-api-access-b278n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:44.319895 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:44.319865 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b278n\" (UniqueName: \"kubernetes.io/projected/935f35f7-28c0-4d05-ab8e-1a2042d7aab4-kube-api-access-b278n\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:44.986652 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:44.986614 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" Oct 30 13:05:44.987156 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:44.986609 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj" event={"ID":"935f35f7-28c0-4d05-ab8e-1a2042d7aab4","Type":"ContainerDied","Data":"fcd872b5d3ad2af50b25808881170c1508589dea47855a015935f1611feb5a1d"} Oct 30 13:05:44.987156 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:44.986789 4413 scope.go:122] "RemoveContainer" containerID="ece04963b3a4e083f12f1f5980b032c461b1e3470c7a472948a3c4f1165247b6" Oct 30 13:05:45.039026 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:45.038999 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj"] Oct 30 13:05:45.051497 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:45.051468 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-734ffe-646746456c-2t5vj"] Oct 30 13:05:45.750781 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:45.750748 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:47.021932 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:47.021903 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="935f35f7-28c0-4d05-ab8e-1a2042d7aab4" path="/var/lib/kubelet/pods/935f35f7-28c0-4d05-ab8e-1a2042d7aab4/volumes" Oct 30 13:05:47.470283 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:47.470234 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:49.054560 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:49.054528 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:50.406281 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:50.405554 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:50.570279 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:50.570223 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7"] Oct 30 13:05:50.729991 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:50.729882 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:50.875099 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:50.875064 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:51.028459 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:51.028355 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:51.254083 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:51.254050 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:51.572712 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:51.572676 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:52.547089 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:52.547053 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:52.679064 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:52.679032 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:53.073668 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:53.073641 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:54.383344 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:54.383311 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z"] Oct 30 13:05:54.383922 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:54.383550 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" podUID="ee65528d-b306-4f62-97fe-6314d1d9660e" containerName="maas-api" containerID="cri-o://28e14e773107a486b04084d5076f540f95110fe5b6f49531ac8bca228265f901" gracePeriod=30 Oct 30 13:05:54.458697 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:54.458665 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:54.681192 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:54.681167 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" Oct 30 13:05:54.774553 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:54.772751 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee65528d-b306-4f62-97fe-6314d1d9660e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee65528d-b306-4f62-97fe-6314d1d9660e-openshift-service-ca\") pod \"ee65528d-b306-4f62-97fe-6314d1d9660e\" (UID: \"ee65528d-b306-4f62-97fe-6314d1d9660e\") " Oct 30 13:05:54.774553 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:54.772855 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee65528d-b306-4f62-97fe-6314d1d9660e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee65528d-b306-4f62-97fe-6314d1d9660e-metrics-certs\") pod \"ee65528d-b306-4f62-97fe-6314d1d9660e\" (UID: \"ee65528d-b306-4f62-97fe-6314d1d9660e\") " Oct 30 13:05:54.774553 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:54.772930 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee65528d-b306-4f62-97fe-6314d1d9660e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee65528d-b306-4f62-97fe-6314d1d9660e-maas-api-tls\") pod \"ee65528d-b306-4f62-97fe-6314d1d9660e\" (UID: \"ee65528d-b306-4f62-97fe-6314d1d9660e\") " Oct 30 13:05:54.774553 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:54.773034 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee65528d-b306-4f62-97fe-6314d1d9660e-kube-api-access-zf4ch\" (UniqueName: \"kubernetes.io/projected/ee65528d-b306-4f62-97fe-6314d1d9660e-kube-api-access-zf4ch\") pod \"ee65528d-b306-4f62-97fe-6314d1d9660e\" (UID: \"ee65528d-b306-4f62-97fe-6314d1d9660e\") " Oct 30 13:05:54.778045 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:54.775354 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee65528d-b306-4f62-97fe-6314d1d9660e-openshift-service-ca" pod "ee65528d-b306-4f62-97fe-6314d1d9660e" (UID: "ee65528d-b306-4f62-97fe-6314d1d9660e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:54.778045 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:54.776961 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee65528d-b306-4f62-97fe-6314d1d9660e-metrics-certs" pod "ee65528d-b306-4f62-97fe-6314d1d9660e" (UID: "ee65528d-b306-4f62-97fe-6314d1d9660e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:54.778712 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:54.778678 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee65528d-b306-4f62-97fe-6314d1d9660e-maas-api-tls" pod "ee65528d-b306-4f62-97fe-6314d1d9660e" (UID: "ee65528d-b306-4f62-97fe-6314d1d9660e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:05:54.778994 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:54.778970 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee65528d-b306-4f62-97fe-6314d1d9660e-kube-api-access-zf4ch" pod "ee65528d-b306-4f62-97fe-6314d1d9660e" (UID: "ee65528d-b306-4f62-97fe-6314d1d9660e"). InnerVolumeSpecName "kube-api-access-zf4ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:54.877133 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:54.875756 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf4ch\" (UniqueName: \"kubernetes.io/projected/ee65528d-b306-4f62-97fe-6314d1d9660e-kube-api-access-zf4ch\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:54.877133 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:54.875789 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee65528d-b306-4f62-97fe-6314d1d9660e-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:54.877133 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:54.875806 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee65528d-b306-4f62-97fe-6314d1d9660e-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:54.877133 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:54.875827 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee65528d-b306-4f62-97fe-6314d1d9660e-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:54.923087 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:54.923005 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:55.078061 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.078029 4413 generic.go:356] "Generic (PLEG): container finished" podID="ee65528d-b306-4f62-97fe-6314d1d9660e" containerID="28e14e773107a486b04084d5076f540f95110fe5b6f49531ac8bca228265f901" exitCode=0 Oct 30 13:05:55.078235 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.078090 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" Oct 30 13:05:55.078235 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.078103 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" event={"ID":"ee65528d-b306-4f62-97fe-6314d1d9660e","Type":"ContainerDied","Data":"28e14e773107a486b04084d5076f540f95110fe5b6f49531ac8bca228265f901"} Oct 30 13:05:55.078235 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.078154 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z" event={"ID":"ee65528d-b306-4f62-97fe-6314d1d9660e","Type":"ContainerDied","Data":"6cd53a6e578acc801f38f6ff11df1ef0ca9d19b7b7159fe9fce266c4141b65e0"} Oct 30 13:05:55.078235 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.078178 4413 scope.go:122] "RemoveContainer" containerID="28e14e773107a486b04084d5076f540f95110fe5b6f49531ac8bca228265f901" Oct 30 13:05:55.088273 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.088219 4413 scope.go:122] "RemoveContainer" containerID="28e14e773107a486b04084d5076f540f95110fe5b6f49531ac8bca228265f901" Oct 30 13:05:55.088578 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:05:55.088553 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e14e773107a486b04084d5076f540f95110fe5b6f49531ac8bca228265f901\": container with ID starting with 28e14e773107a486b04084d5076f540f95110fe5b6f49531ac8bca228265f901 not found: ID does not exist" containerID="28e14e773107a486b04084d5076f540f95110fe5b6f49531ac8bca228265f901" Oct 30 13:05:55.088629 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.088588 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e14e773107a486b04084d5076f540f95110fe5b6f49531ac8bca228265f901"} err="failed to get container status \"28e14e773107a486b04084d5076f540f95110fe5b6f49531ac8bca228265f901\": rpc error: code = NotFound desc = could not find container \"28e14e773107a486b04084d5076f540f95110fe5b6f49531ac8bca228265f901\": container with ID starting with 28e14e773107a486b04084d5076f540f95110fe5b6f49531ac8bca228265f901 not found: ID does not exist" Oct 30 13:05:55.103436 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.103416 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:55.154149 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.154117 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z"] Oct 30 13:05:55.166386 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.166346 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-befaa6-7dd6b7c497-lb52z"] Oct 30 13:05:55.238985 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.238890 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:55.571811 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.571730 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7" podUID="0c2f60ee-cdf2-4a48-a6ff-092f61a349e2" containerName="payload-processing" containerID="cri-o://00a38bdedc007d2daf87be0593ffc9d3230e359c97989704e7b277a9d0a89104" gracePeriod=25 Oct 30 13:05:55.609570 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.609528 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:55.857644 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.857609 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7" Oct 30 13:05:55.987870 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.987836 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2f60ee-cdf2-4a48-a6ff-092f61a349e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2f60ee-cdf2-4a48-a6ff-092f61a349e2-plugins-config-volume\") pod \"0c2f60ee-cdf2-4a48-a6ff-092f61a349e2\" (UID: \"0c2f60ee-cdf2-4a48-a6ff-092f61a349e2\") " Oct 30 13:05:55.988051 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.988024 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2f60ee-cdf2-4a48-a6ff-092f61a349e2-kube-api-access-b86g9\" (UniqueName: \"kubernetes.io/projected/0c2f60ee-cdf2-4a48-a6ff-092f61a349e2-kube-api-access-b86g9\") pod \"0c2f60ee-cdf2-4a48-a6ff-092f61a349e2\" (UID: \"0c2f60ee-cdf2-4a48-a6ff-092f61a349e2\") " Oct 30 13:05:55.988225 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.988197 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2f60ee-cdf2-4a48-a6ff-092f61a349e2-plugins-config-volume" pod "0c2f60ee-cdf2-4a48-a6ff-092f61a349e2" (UID: "0c2f60ee-cdf2-4a48-a6ff-092f61a349e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:05:55.988612 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.988594 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2f60ee-cdf2-4a48-a6ff-092f61a349e2-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:55.990450 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:55.990423 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2f60ee-cdf2-4a48-a6ff-092f61a349e2-kube-api-access-b86g9" pod "0c2f60ee-cdf2-4a48-a6ff-092f61a349e2" (UID: "0c2f60ee-cdf2-4a48-a6ff-092f61a349e2"). InnerVolumeSpecName "kube-api-access-b86g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:05:56.086382 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:56.086301 4413 generic.go:356] "Generic (PLEG): container finished" podID="0c2f60ee-cdf2-4a48-a6ff-092f61a349e2" containerID="00a38bdedc007d2daf87be0593ffc9d3230e359c97989704e7b277a9d0a89104" exitCode=0 Oct 30 13:05:56.086382 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:56.086355 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7" Oct 30 13:05:56.086382 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:56.086369 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7" event={"ID":"0c2f60ee-cdf2-4a48-a6ff-092f61a349e2","Type":"ContainerDied","Data":"00a38bdedc007d2daf87be0593ffc9d3230e359c97989704e7b277a9d0a89104"} Oct 30 13:05:56.086688 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:56.086424 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7" event={"ID":"0c2f60ee-cdf2-4a48-a6ff-092f61a349e2","Type":"ContainerDied","Data":"a5291b5eff714f8632daec0558602e16c6b250de78bd6feff4d50d3ae6164372"} Oct 30 13:05:56.086688 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:56.086452 4413 scope.go:122] "RemoveContainer" containerID="00a38bdedc007d2daf87be0593ffc9d3230e359c97989704e7b277a9d0a89104" Oct 30 13:05:56.089700 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:56.089679 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b86g9\" (UniqueName: \"kubernetes.io/projected/0c2f60ee-cdf2-4a48-a6ff-092f61a349e2-kube-api-access-b86g9\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:05:56.097474 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:56.097453 4413 scope.go:122] "RemoveContainer" containerID="00a38bdedc007d2daf87be0593ffc9d3230e359c97989704e7b277a9d0a89104" Oct 30 13:05:56.097758 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:05:56.097735 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00a38bdedc007d2daf87be0593ffc9d3230e359c97989704e7b277a9d0a89104\": container with ID starting with 00a38bdedc007d2daf87be0593ffc9d3230e359c97989704e7b277a9d0a89104 not found: ID does not exist" containerID="00a38bdedc007d2daf87be0593ffc9d3230e359c97989704e7b277a9d0a89104" Oct 30 13:05:56.097847 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:56.097769 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00a38bdedc007d2daf87be0593ffc9d3230e359c97989704e7b277a9d0a89104"} err="failed to get container status \"00a38bdedc007d2daf87be0593ffc9d3230e359c97989704e7b277a9d0a89104\": rpc error: code = NotFound desc = could not find container \"00a38bdedc007d2daf87be0593ffc9d3230e359c97989704e7b277a9d0a89104\": container with ID starting with 00a38bdedc007d2daf87be0593ffc9d3230e359c97989704e7b277a9d0a89104 not found: ID does not exist" Oct 30 13:05:56.139846 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:56.139815 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7"] Oct 30 13:05:56.142789 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:56.142766 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-73bafc-5bcf696c9b-qlgb7"] Oct 30 13:05:57.021521 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:57.021491 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2f60ee-cdf2-4a48-a6ff-092f61a349e2" path="/var/lib/kubelet/pods/0c2f60ee-cdf2-4a48-a6ff-092f61a349e2/volumes" Oct 30 13:05:57.021961 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:57.021945 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee65528d-b306-4f62-97fe-6314d1d9660e" path="/var/lib/kubelet/pods/ee65528d-b306-4f62-97fe-6314d1d9660e/volumes" Oct 30 13:05:57.175202 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:57.175170 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:58.394294 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:58.394234 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:58.507896 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:58.507866 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:58.756851 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:58.756755 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:05:58.870759 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:05:58.870719 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:00.676869 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:00.676832 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:02.047174 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:02.047141 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:02.166575 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:02.166542 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:02.393612 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:02.393578 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:02.525278 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:02.524423 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:03.183670 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:03.183631 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:04.182496 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:04.182457 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:04.967064 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:04.967028 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:05.816204 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:06:05.816169 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:06:06.063612 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:06.063578 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:07.498998 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:07.498967 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:08.035293 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:08.035233 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:08.182769 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:08.182728 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:08.687790 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:08.687762 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:09.727370 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:09.727342 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:10.103357 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:10.103281 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:10.231920 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:10.231872 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:10.344721 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:10.344692 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:10.460235 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:10.460208 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:10.915430 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:10.915396 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:12.335593 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:12.335538 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:13.594173 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:13.594143 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:13.791350 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:13.791318 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:13.957145 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:13.957106 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:15.422017 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:15.421991 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:15.625604 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:15.625573 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:15.741326 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:15.741224 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:16.064835 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:16.064747 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:16.070746 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:16.070705 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984"] Oct 30 13:06:16.070913 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:16.070891 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" podUID="f9b9b231-747e-45be-8024-7f1b05facd48" containerName="istio-proxy" containerID="cri-o://3a20a90e50605927a6d28c38dbe5075b465e29d260b2742e3f0fd739e00ea3ff" gracePeriod=30 Oct 30 13:06:16.238353 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:16.238311 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:16.673410 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:16.673378 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:16.715178 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:16.715145 4413 patch_prober.go:29] interesting pod/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:06:16.715349 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:16.715204 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" podUID="f9b9b231-747e-45be-8024-7f1b05facd48" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:06:17.834475 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:17.834443 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:18.090775 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:18.090698 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:18.202950 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:18.202909 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:18.428382 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:18.428354 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:19.662499 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:19.662459 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:21.318354 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.318321 4413 generic.go:356] "Generic (PLEG): container finished" podID="f9b9b231-747e-45be-8024-7f1b05facd48" containerID="3a20a90e50605927a6d28c38dbe5075b465e29d260b2742e3f0fd739e00ea3ff" exitCode=0 Oct 30 13:06:21.318817 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.318371 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" event={"ID":"f9b9b231-747e-45be-8024-7f1b05facd48","Type":"ContainerDied","Data":"3a20a90e50605927a6d28c38dbe5075b465e29d260b2742e3f0fd739e00ea3ff"} Oct 30 13:06:21.362665 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.362596 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 13:06:21.479372 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.479333 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b9b231-747e-45be-8024-7f1b05facd48-kube-api-access-p4pgb\" (UniqueName: \"kubernetes.io/projected/f9b9b231-747e-45be-8024-7f1b05facd48-kube-api-access-p4pgb\") pod \"f9b9b231-747e-45be-8024-7f1b05facd48\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " Oct 30 13:06:21.479584 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.479383 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9b9b231-747e-45be-8024-7f1b05facd48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b9b231-747e-45be-8024-7f1b05facd48-istio-podinfo\") pod \"f9b9b231-747e-45be-8024-7f1b05facd48\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " Oct 30 13:06:21.479584 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.479428 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-istio-envoy\") pod \"f9b9b231-747e-45be-8024-7f1b05facd48\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " Oct 30 13:06:21.479584 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.479456 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-workload-socket\") pod \"f9b9b231-747e-45be-8024-7f1b05facd48\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " Oct 30 13:06:21.479584 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.479530 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-istio-data\") pod \"f9b9b231-747e-45be-8024-7f1b05facd48\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " Oct 30 13:06:21.479584 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.479550 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b9b231-747e-45be-8024-7f1b05facd48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b9b231-747e-45be-8024-7f1b05facd48-istiod-ca-cert\") pod \"f9b9b231-747e-45be-8024-7f1b05facd48\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " Oct 30 13:06:21.479584 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.479580 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-workload-certs\") pod \"f9b9b231-747e-45be-8024-7f1b05facd48\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " Oct 30 13:06:21.479921 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.479617 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b9b231-747e-45be-8024-7f1b05facd48-istio-token\" (UniqueName: \"kubernetes.io/projected/f9b9b231-747e-45be-8024-7f1b05facd48-istio-token\") pod \"f9b9b231-747e-45be-8024-7f1b05facd48\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " Oct 30 13:06:21.479921 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.479642 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-credential-socket\") pod \"f9b9b231-747e-45be-8024-7f1b05facd48\" (UID: \"f9b9b231-747e-45be-8024-7f1b05facd48\") " Oct 30 13:06:21.479921 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.479899 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-istio-data" pod "f9b9b231-747e-45be-8024-7f1b05facd48" (UID: "f9b9b231-747e-45be-8024-7f1b05facd48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:21.480141 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.480119 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-workload-socket" pod "f9b9b231-747e-45be-8024-7f1b05facd48" (UID: "f9b9b231-747e-45be-8024-7f1b05facd48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:21.480349 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.480326 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-workload-certs" pod "f9b9b231-747e-45be-8024-7f1b05facd48" (UID: "f9b9b231-747e-45be-8024-7f1b05facd48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:21.480523 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.480503 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-credential-socket" pod "f9b9b231-747e-45be-8024-7f1b05facd48" (UID: "f9b9b231-747e-45be-8024-7f1b05facd48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:21.480645 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.480623 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:21.480645 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.480645 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:21.480645 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.480686 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:21.480915 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.480700 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:21.480915 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.480813 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b9b231-747e-45be-8024-7f1b05facd48-istiod-ca-cert" pod "f9b9b231-747e-45be-8024-7f1b05facd48" (UID: "f9b9b231-747e-45be-8024-7f1b05facd48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:21.482061 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.482033 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b9b231-747e-45be-8024-7f1b05facd48-kube-api-access-p4pgb" pod "f9b9b231-747e-45be-8024-7f1b05facd48" (UID: "f9b9b231-747e-45be-8024-7f1b05facd48"). InnerVolumeSpecName "kube-api-access-p4pgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:21.482141 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.482086 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-istio-envoy" pod "f9b9b231-747e-45be-8024-7f1b05facd48" (UID: "f9b9b231-747e-45be-8024-7f1b05facd48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:06:21.482541 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.482514 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b9b231-747e-45be-8024-7f1b05facd48-istio-token" pod "f9b9b231-747e-45be-8024-7f1b05facd48" (UID: "f9b9b231-747e-45be-8024-7f1b05facd48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:21.483330 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.483309 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9b9b231-747e-45be-8024-7f1b05facd48-istio-podinfo" pod "f9b9b231-747e-45be-8024-7f1b05facd48" (UID: "f9b9b231-747e-45be-8024-7f1b05facd48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:06:21.552686 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.552652 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:21.582517 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.582484 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b9b231-747e-45be-8024-7f1b05facd48-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:21.582517 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.582510 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b9b231-747e-45be-8024-7f1b05facd48-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:21.582517 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.582519 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b9b231-747e-45be-8024-7f1b05facd48-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:21.582808 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.582530 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4pgb\" (UniqueName: \"kubernetes.io/projected/f9b9b231-747e-45be-8024-7f1b05facd48-kube-api-access-p4pgb\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:21.582808 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:21.582540 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b9b231-747e-45be-8024-7f1b05facd48-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:22.326852 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:22.326822 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" Oct 30 13:06:22.327322 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:22.326856 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984" event={"ID":"f9b9b231-747e-45be-8024-7f1b05facd48","Type":"ContainerDied","Data":"9e46d20dce12a20345352bcf6f0a13f85e8b404a2e25fbeee85c83203254ce63"} Oct 30 13:06:22.327322 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:22.326918 4413 scope.go:122] "RemoveContainer" containerID="3a20a90e50605927a6d28c38dbe5075b465e29d260b2742e3f0fd739e00ea3ff" Oct 30 13:06:22.391641 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:22.391602 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984"] Oct 30 13:06:22.403489 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:22.403458 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c1b1e8-openshift-default-6c6975cfc9-jr984"] Oct 30 13:06:23.023186 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:23.023150 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b9b231-747e-45be-8024-7f1b05facd48" path="/var/lib/kubelet/pods/f9b9b231-747e-45be-8024-7f1b05facd48/volumes" Oct 30 13:06:23.377103 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:23.377068 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:23.672878 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:23.672800 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:23.824845 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:23.824813 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:24.059802 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:24.059723 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:25.068753 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:25.068724 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:26.559235 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:26.559193 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk"] Oct 30 13:06:26.559786 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:26.559463 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" podUID="27ef9a37-c090-4cbb-824f-f0a9c54d3fe2" containerName="maas-api" containerID="cri-o://09be7f1a5b4dcc460e7974aefdd4d5128c2edc9e9c51f5886cbd5424a74900c3" gracePeriod=30 Oct 30 13:06:26.643263 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:26.643212 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:26.833165 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:26.833146 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 13:06:26.963675 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:26.963649 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-openshift-service-ca\") pod \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\" (UID: \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\") " Oct 30 13:06:26.963866 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:26.963765 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-metrics-certs\") pod \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\" (UID: \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\") " Oct 30 13:06:26.963866 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:26.963790 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-maas-api-tls\") pod \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\" (UID: \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\") " Oct 30 13:06:26.964008 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:26.963957 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-kube-api-access-ptj7s\" (UniqueName: \"kubernetes.io/projected/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-kube-api-access-ptj7s\") pod \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\" (UID: \"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2\") " Oct 30 13:06:26.964094 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:26.964064 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-openshift-service-ca" pod "27ef9a37-c090-4cbb-824f-f0a9c54d3fe2" (UID: "27ef9a37-c090-4cbb-824f-f0a9c54d3fe2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:26.964540 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:26.964474 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:26.966174 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:26.966151 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-maas-api-tls" pod "27ef9a37-c090-4cbb-824f-f0a9c54d3fe2" (UID: "27ef9a37-c090-4cbb-824f-f0a9c54d3fe2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:26.966450 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:26.966433 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-metrics-certs" pod "27ef9a37-c090-4cbb-824f-f0a9c54d3fe2" (UID: "27ef9a37-c090-4cbb-824f-f0a9c54d3fe2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:26.966618 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:26.966596 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-kube-api-access-ptj7s" pod "27ef9a37-c090-4cbb-824f-f0a9c54d3fe2" (UID: "27ef9a37-c090-4cbb-824f-f0a9c54d3fe2"). InnerVolumeSpecName "kube-api-access-ptj7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:27.065523 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:27.065461 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptj7s\" (UniqueName: \"kubernetes.io/projected/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-kube-api-access-ptj7s\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:27.065523 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:27.065490 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:27.065523 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:27.065501 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:27.295000 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:27.294965 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:27.359930 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:27.359844 4413 generic.go:356] "Generic (PLEG): container finished" podID="27ef9a37-c090-4cbb-824f-f0a9c54d3fe2" containerID="09be7f1a5b4dcc460e7974aefdd4d5128c2edc9e9c51f5886cbd5424a74900c3" exitCode=0 Oct 30 13:06:27.359930 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:27.359897 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" Oct 30 13:06:27.360131 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:27.359927 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" event={"ID":"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2","Type":"ContainerDied","Data":"09be7f1a5b4dcc460e7974aefdd4d5128c2edc9e9c51f5886cbd5424a74900c3"} Oct 30 13:06:27.360131 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:27.359978 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk" event={"ID":"27ef9a37-c090-4cbb-824f-f0a9c54d3fe2","Type":"ContainerDied","Data":"24fec8daa20eab865c9d72f9d2db960b800267bf2821961e49a7610582bb181a"} Oct 30 13:06:27.360131 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:27.359999 4413 scope.go:122] "RemoveContainer" containerID="09be7f1a5b4dcc460e7974aefdd4d5128c2edc9e9c51f5886cbd5424a74900c3" Oct 30 13:06:27.370597 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:27.370574 4413 scope.go:122] "RemoveContainer" containerID="09be7f1a5b4dcc460e7974aefdd4d5128c2edc9e9c51f5886cbd5424a74900c3" Oct 30 13:06:27.370884 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:06:27.370865 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09be7f1a5b4dcc460e7974aefdd4d5128c2edc9e9c51f5886cbd5424a74900c3\": container with ID starting with 09be7f1a5b4dcc460e7974aefdd4d5128c2edc9e9c51f5886cbd5424a74900c3 not found: ID does not exist" containerID="09be7f1a5b4dcc460e7974aefdd4d5128c2edc9e9c51f5886cbd5424a74900c3" Oct 30 13:06:27.370944 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:27.370893 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09be7f1a5b4dcc460e7974aefdd4d5128c2edc9e9c51f5886cbd5424a74900c3"} err="failed to get container status \"09be7f1a5b4dcc460e7974aefdd4d5128c2edc9e9c51f5886cbd5424a74900c3\": rpc error: code = NotFound desc = could not find container \"09be7f1a5b4dcc460e7974aefdd4d5128c2edc9e9c51f5886cbd5424a74900c3\": container with ID starting with 09be7f1a5b4dcc460e7974aefdd4d5128c2edc9e9c51f5886cbd5424a74900c3 not found: ID does not exist" Oct 30 13:06:27.421896 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:27.421864 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk"] Oct 30 13:06:27.429115 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:27.429092 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a786af-6b65c44d94-4s4dk"] Oct 30 13:06:27.436149 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:27.436096 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:27.557067 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:27.557032 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:27.901945 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:27.901919 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:29.020319 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:29.020287 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ef9a37-c090-4cbb-824f-f0a9c54d3fe2" path="/var/lib/kubelet/pods/27ef9a37-c090-4cbb-824f-f0a9c54d3fe2/volumes" Oct 30 13:06:29.257091 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:29.257056 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:29.392213 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:29.392180 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j"] Oct 30 13:06:29.427985 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:29.427955 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k"] Oct 30 13:06:29.562122 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:29.562093 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:29.683897 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:29.683819 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:30.948148 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:30.948109 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:32.639868 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:32.639834 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:33.993656 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:33.993603 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc"] Oct 30 13:06:33.994164 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:33.993914 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" podUID="37eb395b-d510-4d1c-a59c-50d5d42b0efd" containerName="maas-api" containerID="cri-o://765523985b3c8a0284ed58859f02613a305d56bf51fcdaf76e40b6baee5e17c5" gracePeriod=30 Oct 30 13:06:34.057089 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.057059 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:34.285105 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.285084 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" Oct 30 13:06:34.346602 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.346576 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37eb395b-d510-4d1c-a59c-50d5d42b0efd-kube-api-access-8v8qm\" (UniqueName: \"kubernetes.io/projected/37eb395b-d510-4d1c-a59c-50d5d42b0efd-kube-api-access-8v8qm\") pod \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\" (UID: \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\") " Oct 30 13:06:34.346775 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.346662 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37eb395b-d510-4d1c-a59c-50d5d42b0efd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37eb395b-d510-4d1c-a59c-50d5d42b0efd-openshift-service-ca\") pod \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\" (UID: \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\") " Oct 30 13:06:34.346866 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.346840 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37eb395b-d510-4d1c-a59c-50d5d42b0efd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/37eb395b-d510-4d1c-a59c-50d5d42b0efd-metrics-certs\") pod \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\" (UID: \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\") " Oct 30 13:06:34.346991 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.346968 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37eb395b-d510-4d1c-a59c-50d5d42b0efd-openshift-service-ca" pod "37eb395b-d510-4d1c-a59c-50d5d42b0efd" (UID: "37eb395b-d510-4d1c-a59c-50d5d42b0efd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:34.347086 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.346987 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37eb395b-d510-4d1c-a59c-50d5d42b0efd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37eb395b-d510-4d1c-a59c-50d5d42b0efd-maas-api-tls\") pod \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\" (UID: \"37eb395b-d510-4d1c-a59c-50d5d42b0efd\") " Oct 30 13:06:34.347687 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.347659 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37eb395b-d510-4d1c-a59c-50d5d42b0efd-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:34.349116 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.349092 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37eb395b-d510-4d1c-a59c-50d5d42b0efd-metrics-certs" pod "37eb395b-d510-4d1c-a59c-50d5d42b0efd" (UID: "37eb395b-d510-4d1c-a59c-50d5d42b0efd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:34.349423 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.349402 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37eb395b-d510-4d1c-a59c-50d5d42b0efd-kube-api-access-8v8qm" pod "37eb395b-d510-4d1c-a59c-50d5d42b0efd" (UID: "37eb395b-d510-4d1c-a59c-50d5d42b0efd"). InnerVolumeSpecName "kube-api-access-8v8qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:34.349576 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.349526 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37eb395b-d510-4d1c-a59c-50d5d42b0efd-maas-api-tls" pod "37eb395b-d510-4d1c-a59c-50d5d42b0efd" (UID: "37eb395b-d510-4d1c-a59c-50d5d42b0efd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:06:34.355410 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.355384 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:34.393119 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.393083 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j" podUID="b770b1fb-b8b3-4f50-93ea-9e88d2e89922" containerName="payload-processing" containerID="cri-o://69f96fc525f89e08940c8313a54ef0ce60ac157e78204b5641e29e74ba8ddaac" gracePeriod=25 Oct 30 13:06:34.410838 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.410806 4413 generic.go:356] "Generic (PLEG): container finished" podID="37eb395b-d510-4d1c-a59c-50d5d42b0efd" containerID="765523985b3c8a0284ed58859f02613a305d56bf51fcdaf76e40b6baee5e17c5" exitCode=0 Oct 30 13:06:34.410969 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.410861 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" Oct 30 13:06:34.410969 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.410893 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" event={"ID":"37eb395b-d510-4d1c-a59c-50d5d42b0efd","Type":"ContainerDied","Data":"765523985b3c8a0284ed58859f02613a305d56bf51fcdaf76e40b6baee5e17c5"} Oct 30 13:06:34.410969 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.410960 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc" event={"ID":"37eb395b-d510-4d1c-a59c-50d5d42b0efd","Type":"ContainerDied","Data":"93d78b0f3befab5a89b16e8402a3152ffe2b9438f40b1d0943d90c17f571f567"} Oct 30 13:06:34.411122 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.410987 4413 scope.go:122] "RemoveContainer" containerID="765523985b3c8a0284ed58859f02613a305d56bf51fcdaf76e40b6baee5e17c5" Oct 30 13:06:34.428808 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.428783 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k" podUID="d7a63529-8b71-40df-b440-d8b5316d112a" containerName="payload-pre-processing" containerID="cri-o://a1a8327bcf340bdb0916af6950688d3a6a585a88abd92f20a0180bc3419f096f" gracePeriod=25 Oct 30 13:06:34.449556 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.448975 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37eb395b-d510-4d1c-a59c-50d5d42b0efd-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:34.449556 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.449010 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37eb395b-d510-4d1c-a59c-50d5d42b0efd-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:34.449556 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.449025 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v8qm\" (UniqueName: \"kubernetes.io/projected/37eb395b-d510-4d1c-a59c-50d5d42b0efd-kube-api-access-8v8qm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:34.481276 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.479322 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:34.512152 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.512129 4413 scope.go:122] "RemoveContainer" containerID="765523985b3c8a0284ed58859f02613a305d56bf51fcdaf76e40b6baee5e17c5" Oct 30 13:06:34.512506 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:06:34.512485 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765523985b3c8a0284ed58859f02613a305d56bf51fcdaf76e40b6baee5e17c5\": container with ID starting with 765523985b3c8a0284ed58859f02613a305d56bf51fcdaf76e40b6baee5e17c5 not found: ID does not exist" containerID="765523985b3c8a0284ed58859f02613a305d56bf51fcdaf76e40b6baee5e17c5" Oct 30 13:06:34.512604 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.512513 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765523985b3c8a0284ed58859f02613a305d56bf51fcdaf76e40b6baee5e17c5"} err="failed to get container status \"765523985b3c8a0284ed58859f02613a305d56bf51fcdaf76e40b6baee5e17c5\": rpc error: code = NotFound desc = could not find container \"765523985b3c8a0284ed58859f02613a305d56bf51fcdaf76e40b6baee5e17c5\": container with ID starting with 765523985b3c8a0284ed58859f02613a305d56bf51fcdaf76e40b6baee5e17c5 not found: ID does not exist" Oct 30 13:06:34.534350 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.534319 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc"] Oct 30 13:06:34.538945 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.538923 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2545c7-d655f658c-9g7kc"] Oct 30 13:06:34.613747 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.613708 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:34.696038 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.696006 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j" Oct 30 13:06:34.746713 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.746678 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:34.748760 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.748735 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k" Oct 30 13:06:34.753481 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.753409 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b770b1fb-b8b3-4f50-93ea-9e88d2e89922-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b770b1fb-b8b3-4f50-93ea-9e88d2e89922-plugins-config-volume\") pod \"b770b1fb-b8b3-4f50-93ea-9e88d2e89922\" (UID: \"b770b1fb-b8b3-4f50-93ea-9e88d2e89922\") " Oct 30 13:06:34.753730 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.753709 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b770b1fb-b8b3-4f50-93ea-9e88d2e89922-plugins-config-volume" pod "b770b1fb-b8b3-4f50-93ea-9e88d2e89922" (UID: "b770b1fb-b8b3-4f50-93ea-9e88d2e89922"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:34.753813 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.753747 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b770b1fb-b8b3-4f50-93ea-9e88d2e89922-kube-api-access-ll2q9\" (UniqueName: \"kubernetes.io/projected/b770b1fb-b8b3-4f50-93ea-9e88d2e89922-kube-api-access-ll2q9\") pod \"b770b1fb-b8b3-4f50-93ea-9e88d2e89922\" (UID: \"b770b1fb-b8b3-4f50-93ea-9e88d2e89922\") " Oct 30 13:06:34.754239 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.754217 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b770b1fb-b8b3-4f50-93ea-9e88d2e89922-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:34.756037 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.756010 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b770b1fb-b8b3-4f50-93ea-9e88d2e89922-kube-api-access-ll2q9" pod "b770b1fb-b8b3-4f50-93ea-9e88d2e89922" (UID: "b770b1fb-b8b3-4f50-93ea-9e88d2e89922"). InnerVolumeSpecName "kube-api-access-ll2q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:34.854709 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.854624 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7a63529-8b71-40df-b440-d8b5316d112a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a63529-8b71-40df-b440-d8b5316d112a-plugins-config-volume\") pod \"d7a63529-8b71-40df-b440-d8b5316d112a\" (UID: \"d7a63529-8b71-40df-b440-d8b5316d112a\") " Oct 30 13:06:34.854709 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.854657 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7a63529-8b71-40df-b440-d8b5316d112a-kube-api-access-bqzpm\" (UniqueName: \"kubernetes.io/projected/d7a63529-8b71-40df-b440-d8b5316d112a-kube-api-access-bqzpm\") pod \"d7a63529-8b71-40df-b440-d8b5316d112a\" (UID: \"d7a63529-8b71-40df-b440-d8b5316d112a\") " Oct 30 13:06:34.855020 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.854994 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7a63529-8b71-40df-b440-d8b5316d112a-plugins-config-volume" pod "d7a63529-8b71-40df-b440-d8b5316d112a" (UID: "d7a63529-8b71-40df-b440-d8b5316d112a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:34.855146 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.855126 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a63529-8b71-40df-b440-d8b5316d112a-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:34.855194 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.855145 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll2q9\" (UniqueName: \"kubernetes.io/projected/b770b1fb-b8b3-4f50-93ea-9e88d2e89922-kube-api-access-ll2q9\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:34.857043 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.857017 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a63529-8b71-40df-b440-d8b5316d112a-kube-api-access-bqzpm" pod "d7a63529-8b71-40df-b440-d8b5316d112a" (UID: "d7a63529-8b71-40df-b440-d8b5316d112a"). InnerVolumeSpecName "kube-api-access-bqzpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:34.956165 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:34.956140 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqzpm\" (UniqueName: \"kubernetes.io/projected/d7a63529-8b71-40df-b440-d8b5316d112a-kube-api-access-bqzpm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:35.021869 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.021840 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37eb395b-d510-4d1c-a59c-50d5d42b0efd" path="/var/lib/kubelet/pods/37eb395b-d510-4d1c-a59c-50d5d42b0efd/volumes" Oct 30 13:06:35.314945 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.314918 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:35.419938 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.419905 4413 generic.go:356] "Generic (PLEG): container finished" podID="d7a63529-8b71-40df-b440-d8b5316d112a" containerID="a1a8327bcf340bdb0916af6950688d3a6a585a88abd92f20a0180bc3419f096f" exitCode=0 Oct 30 13:06:35.420111 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.419965 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k" Oct 30 13:06:35.420111 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.419991 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k" event={"ID":"d7a63529-8b71-40df-b440-d8b5316d112a","Type":"ContainerDied","Data":"a1a8327bcf340bdb0916af6950688d3a6a585a88abd92f20a0180bc3419f096f"} Oct 30 13:06:35.420111 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.420044 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k" event={"ID":"d7a63529-8b71-40df-b440-d8b5316d112a","Type":"ContainerDied","Data":"1ae1497293bc3ec044495c975fef115403cc691c19e323391a4ab1ceca2469f2"} Oct 30 13:06:35.420111 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.420067 4413 scope.go:122] "RemoveContainer" containerID="a1a8327bcf340bdb0916af6950688d3a6a585a88abd92f20a0180bc3419f096f" Oct 30 13:06:35.421679 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.421656 4413 generic.go:356] "Generic (PLEG): container finished" podID="b770b1fb-b8b3-4f50-93ea-9e88d2e89922" containerID="69f96fc525f89e08940c8313a54ef0ce60ac157e78204b5641e29e74ba8ddaac" exitCode=0 Oct 30 13:06:35.421766 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.421691 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j" event={"ID":"b770b1fb-b8b3-4f50-93ea-9e88d2e89922","Type":"ContainerDied","Data":"69f96fc525f89e08940c8313a54ef0ce60ac157e78204b5641e29e74ba8ddaac"} Oct 30 13:06:35.421766 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.421717 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j" Oct 30 13:06:35.421766 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.421730 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j" event={"ID":"b770b1fb-b8b3-4f50-93ea-9e88d2e89922","Type":"ContainerDied","Data":"6fefec6c253125d41727469418a801cef2ec1b72e219d2e75ace4c4282c2f00a"} Oct 30 13:06:35.430370 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.430350 4413 scope.go:122] "RemoveContainer" containerID="a1a8327bcf340bdb0916af6950688d3a6a585a88abd92f20a0180bc3419f096f" Oct 30 13:06:35.430656 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:06:35.430638 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1a8327bcf340bdb0916af6950688d3a6a585a88abd92f20a0180bc3419f096f\": container with ID starting with a1a8327bcf340bdb0916af6950688d3a6a585a88abd92f20a0180bc3419f096f not found: ID does not exist" containerID="a1a8327bcf340bdb0916af6950688d3a6a585a88abd92f20a0180bc3419f096f" Oct 30 13:06:35.430732 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.430662 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a8327bcf340bdb0916af6950688d3a6a585a88abd92f20a0180bc3419f096f"} err="failed to get container status \"a1a8327bcf340bdb0916af6950688d3a6a585a88abd92f20a0180bc3419f096f\": rpc error: code = NotFound desc = could not find container \"a1a8327bcf340bdb0916af6950688d3a6a585a88abd92f20a0180bc3419f096f\": container with ID starting with a1a8327bcf340bdb0916af6950688d3a6a585a88abd92f20a0180bc3419f096f not found: ID does not exist" Oct 30 13:06:35.430732 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.430680 4413 scope.go:122] "RemoveContainer" containerID="69f96fc525f89e08940c8313a54ef0ce60ac157e78204b5641e29e74ba8ddaac" Oct 30 13:06:35.439623 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.439602 4413 scope.go:122] "RemoveContainer" containerID="69f96fc525f89e08940c8313a54ef0ce60ac157e78204b5641e29e74ba8ddaac" Oct 30 13:06:35.439894 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:06:35.439876 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69f96fc525f89e08940c8313a54ef0ce60ac157e78204b5641e29e74ba8ddaac\": container with ID starting with 69f96fc525f89e08940c8313a54ef0ce60ac157e78204b5641e29e74ba8ddaac not found: ID does not exist" containerID="69f96fc525f89e08940c8313a54ef0ce60ac157e78204b5641e29e74ba8ddaac" Oct 30 13:06:35.439961 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.439902 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69f96fc525f89e08940c8313a54ef0ce60ac157e78204b5641e29e74ba8ddaac"} err="failed to get container status \"69f96fc525f89e08940c8313a54ef0ce60ac157e78204b5641e29e74ba8ddaac\": rpc error: code = NotFound desc = could not find container \"69f96fc525f89e08940c8313a54ef0ce60ac157e78204b5641e29e74ba8ddaac\": container with ID starting with 69f96fc525f89e08940c8313a54ef0ce60ac157e78204b5641e29e74ba8ddaac not found: ID does not exist" Oct 30 13:06:35.479335 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.479307 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j"] Oct 30 13:06:35.490511 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.490491 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dd0628-996645978-zqv8j"] Oct 30 13:06:35.519014 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.518993 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k"] Oct 30 13:06:35.523275 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.523236 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dd0628-8484bd5cbc-8sq4k"] Oct 30 13:06:35.674304 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.674276 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc"] Oct 30 13:06:35.804441 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.804400 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:35.814679 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:06:35.814643 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:06:35.919483 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:35.919454 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:36.554177 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:36.554146 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:37.020917 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:37.020886 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b770b1fb-b8b3-4f50-93ea-9e88d2e89922" path="/var/lib/kubelet/pods/b770b1fb-b8b3-4f50-93ea-9e88d2e89922/volumes" Oct 30 13:06:37.021371 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:37.021356 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7a63529-8b71-40df-b440-d8b5316d112a" path="/var/lib/kubelet/pods/d7a63529-8b71-40df-b440-d8b5316d112a/volumes" Oct 30 13:06:38.155871 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:38.155839 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:39.845743 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:39.845714 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:40.675185 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:40.675152 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc" podUID="ac48ad61-b11f-46c7-902e-87fa0d893933" containerName="payload-processing" containerID="cri-o://6c9a85cc056d85e6a87aa81bee98afd5cfd493b57d9e9daccb35ec4900514f23" gracePeriod=25 Oct 30 13:06:40.945562 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:40.945506 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc" Oct 30 13:06:41.034113 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:41.034087 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac48ad61-b11f-46c7-902e-87fa0d893933-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac48ad61-b11f-46c7-902e-87fa0d893933-plugins-config-volume\") pod \"ac48ad61-b11f-46c7-902e-87fa0d893933\" (UID: \"ac48ad61-b11f-46c7-902e-87fa0d893933\") " Oct 30 13:06:41.034323 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:41.034204 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac48ad61-b11f-46c7-902e-87fa0d893933-kube-api-access-xsb9m\" (UniqueName: \"kubernetes.io/projected/ac48ad61-b11f-46c7-902e-87fa0d893933-kube-api-access-xsb9m\") pod \"ac48ad61-b11f-46c7-902e-87fa0d893933\" (UID: \"ac48ad61-b11f-46c7-902e-87fa0d893933\") " Oct 30 13:06:41.034532 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:41.034509 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac48ad61-b11f-46c7-902e-87fa0d893933-plugins-config-volume" pod "ac48ad61-b11f-46c7-902e-87fa0d893933" (UID: "ac48ad61-b11f-46c7-902e-87fa0d893933"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:06:41.034778 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:41.034761 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac48ad61-b11f-46c7-902e-87fa0d893933-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:41.036444 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:41.036419 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac48ad61-b11f-46c7-902e-87fa0d893933-kube-api-access-xsb9m" pod "ac48ad61-b11f-46c7-902e-87fa0d893933" (UID: "ac48ad61-b11f-46c7-902e-87fa0d893933"). InnerVolumeSpecName "kube-api-access-xsb9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:06:41.137884 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:41.137858 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsb9m\" (UniqueName: \"kubernetes.io/projected/ac48ad61-b11f-46c7-902e-87fa0d893933-kube-api-access-xsb9m\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:06:41.462887 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:41.462855 4413 generic.go:356] "Generic (PLEG): container finished" podID="ac48ad61-b11f-46c7-902e-87fa0d893933" containerID="6c9a85cc056d85e6a87aa81bee98afd5cfd493b57d9e9daccb35ec4900514f23" exitCode=0 Oct 30 13:06:41.463057 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:41.462923 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc" Oct 30 13:06:41.463057 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:41.462918 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc" event={"ID":"ac48ad61-b11f-46c7-902e-87fa0d893933","Type":"ContainerDied","Data":"6c9a85cc056d85e6a87aa81bee98afd5cfd493b57d9e9daccb35ec4900514f23"} Oct 30 13:06:41.463057 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:41.463014 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc" event={"ID":"ac48ad61-b11f-46c7-902e-87fa0d893933","Type":"ContainerDied","Data":"5589dd93ded9222cfd4f598a216ae7e6fed6bcef1b4044692bf2b99309839517"} Oct 30 13:06:41.463057 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:41.463044 4413 scope.go:122] "RemoveContainer" containerID="6c9a85cc056d85e6a87aa81bee98afd5cfd493b57d9e9daccb35ec4900514f23" Oct 30 13:06:41.474734 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:41.474715 4413 scope.go:122] "RemoveContainer" containerID="6c9a85cc056d85e6a87aa81bee98afd5cfd493b57d9e9daccb35ec4900514f23" Oct 30 13:06:41.475000 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:06:41.474978 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9a85cc056d85e6a87aa81bee98afd5cfd493b57d9e9daccb35ec4900514f23\": container with ID starting with 6c9a85cc056d85e6a87aa81bee98afd5cfd493b57d9e9daccb35ec4900514f23 not found: ID does not exist" containerID="6c9a85cc056d85e6a87aa81bee98afd5cfd493b57d9e9daccb35ec4900514f23" Oct 30 13:06:41.475048 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:41.475008 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9a85cc056d85e6a87aa81bee98afd5cfd493b57d9e9daccb35ec4900514f23"} err="failed to get container status \"6c9a85cc056d85e6a87aa81bee98afd5cfd493b57d9e9daccb35ec4900514f23\": rpc error: code = NotFound desc = could not find container \"6c9a85cc056d85e6a87aa81bee98afd5cfd493b57d9e9daccb35ec4900514f23\": container with ID starting with 6c9a85cc056d85e6a87aa81bee98afd5cfd493b57d9e9daccb35ec4900514f23 not found: ID does not exist" Oct 30 13:06:41.515435 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:41.515413 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc"] Oct 30 13:06:41.519918 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:41.519899 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c3f12-7f464c7b8b-25xkc"] Oct 30 13:06:41.525439 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:41.525403 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:41.634879 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:41.634840 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:41.867018 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:41.866947 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:43.020057 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:43.020029 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac48ad61-b11f-46c7-902e-87fa0d893933" path="/var/lib/kubelet/pods/ac48ad61-b11f-46c7-902e-87fa0d893933/volumes" Oct 30 13:06:43.418431 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:43.418380 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:45.308799 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:45.308764 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:45.863876 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:45.863839 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg"] Oct 30 13:06:46.074803 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:46.074767 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:46.242195 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:46.242166 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:47.466680 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:47.466645 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:49.122738 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:49.122660 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:50.604739 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:50.604706 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:50.872325 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:50.871326 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" podUID="29b11889-7268-40a1-8387-109af1464e04" containerName="payload-processing" containerID="cri-o://e4453e893052b35169aedbd5791044a806d3a6a955ab98e47474d68f7bbb3f64" gracePeriod=25 Oct 30 13:06:50.900534 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:50.900507 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:51.027068 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:51.027041 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:51.147175 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:51.147136 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:51.327045 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:51.327002 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" podUID="29b11889-7268-40a1-8387-109af1464e04" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:06:51.534011 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:51.533930 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:51.856171 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:51.856087 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:52.265074 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:52.265038 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:52.877777 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:52.877746 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb"] Oct 30 13:06:53.044576 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:53.044544 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:53.160358 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:53.160237 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:53.667739 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:53.667626 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:55.360136 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:55.360107 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:57.005895 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:57.005866 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:57.878060 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:57.878002 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" podUID="835438be-d2d1-40da-a7ed-cd420db69319" containerName="payload-processing" containerID="cri-o://f71d1c54fdd9603db20a4ee568489b69d82a344bd7de4b3051819e482d46d01c" gracePeriod=25 Oct 30 13:06:58.566334 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:58.566298 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:58.692181 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:58.692148 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:58.940429 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:58.940399 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:59.057136 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:59.057104 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:06:59.451893 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:06:59.451864 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:00.090382 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:00.090353 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:00.577985 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:00.577953 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:00.711723 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:00.711686 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:01.210513 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:01.210483 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:01.326908 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:01.326869 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" podUID="29b11889-7268-40a1-8387-109af1464e04" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:02.413659 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:02.413621 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" podUID="835438be-d2d1-40da-a7ed-cd420db69319" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:02.610809 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:02.610773 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:03.314670 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:03.314636 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:04.344309 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:04.344279 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:04.412033 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:04.412003 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j"] Oct 30 13:07:04.537647 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:04.537571 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:05.714923 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:05.714892 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:05.813378 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:07:05.813344 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:07:06.238132 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:06.238095 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:06.351091 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:06.351058 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:06.464733 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:06.464706 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:06.810294 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:06.810260 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:08.377302 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:08.377232 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:08.698491 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:08.698405 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:08.815636 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:08.815601 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:08.964162 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:08.964082 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:09.899230 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:09.899196 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d"] Oct 30 13:07:09.946859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:09.946827 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv"] Oct 30 13:07:10.082820 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:10.082791 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:10.239408 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:10.239332 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:10.362051 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:10.362010 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:10.982142 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:10.982106 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:11.107116 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:11.107084 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:11.160834 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:11.160813 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" Oct 30 13:07:11.234878 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:11.234849 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b11889-7268-40a1-8387-109af1464e04-kube-api-access-pkvc8\" (UniqueName: \"kubernetes.io/projected/29b11889-7268-40a1-8387-109af1464e04-kube-api-access-pkvc8\") pod \"29b11889-7268-40a1-8387-109af1464e04\" (UID: \"29b11889-7268-40a1-8387-109af1464e04\") " Oct 30 13:07:11.235066 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:11.235051 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b11889-7268-40a1-8387-109af1464e04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b11889-7268-40a1-8387-109af1464e04-plugins-config-volume\") pod \"29b11889-7268-40a1-8387-109af1464e04\" (UID: \"29b11889-7268-40a1-8387-109af1464e04\") " Oct 30 13:07:11.235539 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:11.235515 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b11889-7268-40a1-8387-109af1464e04-plugins-config-volume" pod "29b11889-7268-40a1-8387-109af1464e04" (UID: "29b11889-7268-40a1-8387-109af1464e04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:11.237202 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:11.237182 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b11889-7268-40a1-8387-109af1464e04-kube-api-access-pkvc8" pod "29b11889-7268-40a1-8387-109af1464e04" (UID: "29b11889-7268-40a1-8387-109af1464e04"). InnerVolumeSpecName "kube-api-access-pkvc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:11.336888 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:11.336862 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b11889-7268-40a1-8387-109af1464e04-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:11.337026 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:11.336892 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkvc8\" (UniqueName: \"kubernetes.io/projected/29b11889-7268-40a1-8387-109af1464e04-kube-api-access-pkvc8\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:11.674375 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:11.674346 4413 generic.go:356] "Generic (PLEG): container finished" podID="29b11889-7268-40a1-8387-109af1464e04" containerID="e4453e893052b35169aedbd5791044a806d3a6a955ab98e47474d68f7bbb3f64" exitCode=0 Oct 30 13:07:11.674581 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:11.674404 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" Oct 30 13:07:11.674581 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:11.674423 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" event={"ID":"29b11889-7268-40a1-8387-109af1464e04","Type":"ContainerDied","Data":"e4453e893052b35169aedbd5791044a806d3a6a955ab98e47474d68f7bbb3f64"} Oct 30 13:07:11.674581 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:11.674463 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg" event={"ID":"29b11889-7268-40a1-8387-109af1464e04","Type":"ContainerDied","Data":"25f001ebe36d067c6a712645341747d6a994e92be02aeaf5f2136239763a66b1"} Oct 30 13:07:11.674581 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:11.674483 4413 scope.go:122] "RemoveContainer" containerID="e4453e893052b35169aedbd5791044a806d3a6a955ab98e47474d68f7bbb3f64" Oct 30 13:07:11.686320 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:11.686293 4413 scope.go:122] "RemoveContainer" containerID="e4453e893052b35169aedbd5791044a806d3a6a955ab98e47474d68f7bbb3f64" Oct 30 13:07:11.686581 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:07:11.686562 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4453e893052b35169aedbd5791044a806d3a6a955ab98e47474d68f7bbb3f64\": container with ID starting with e4453e893052b35169aedbd5791044a806d3a6a955ab98e47474d68f7bbb3f64 not found: ID does not exist" containerID="e4453e893052b35169aedbd5791044a806d3a6a955ab98e47474d68f7bbb3f64" Oct 30 13:07:11.686632 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:11.686587 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4453e893052b35169aedbd5791044a806d3a6a955ab98e47474d68f7bbb3f64"} err="failed to get container status \"e4453e893052b35169aedbd5791044a806d3a6a955ab98e47474d68f7bbb3f64\": rpc error: code = NotFound desc = could not find container \"e4453e893052b35169aedbd5791044a806d3a6a955ab98e47474d68f7bbb3f64\": container with ID starting with e4453e893052b35169aedbd5791044a806d3a6a955ab98e47474d68f7bbb3f64 not found: ID does not exist" Oct 30 13:07:11.725741 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:11.725718 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg"] Oct 30 13:07:11.737623 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:11.737585 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0f7d14-77597d868b-sw2sg"] Oct 30 13:07:12.098110 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:12.098031 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:12.414364 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:12.414324 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" podUID="835438be-d2d1-40da-a7ed-cd420db69319" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:12.459978 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:12.459954 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:13.021832 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:13.021797 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b11889-7268-40a1-8387-109af1464e04" path="/var/lib/kubelet/pods/29b11889-7268-40a1-8387-109af1464e04/volumes" Oct 30 13:07:13.758025 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:13.757989 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:13.889784 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:13.889756 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:14.123022 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:14.122945 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:14.900532 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:14.900498 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d" podUID="4295a82c-12f5-4451-b82a-0203044f1de3" containerName="payload-processing" containerID="cri-o://fa5c880995796b92034c6b319bc660b4b1d6bec9bc52c5e4cf5d1da08642ecc3" gracePeriod=25 Oct 30 13:07:14.947553 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:14.947517 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv" podUID="ae223397-5523-4836-9d6c-7e2daf2ae313" containerName="payload-pre-processing" containerID="cri-o://6ce3aa07d7c96003104c6fff246ec2596986c6b06b7fe4afe4ce36a5abbf4d39" gracePeriod=25 Oct 30 13:07:15.213259 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.213218 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d" Oct 30 13:07:15.250944 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.250920 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv" Oct 30 13:07:15.300590 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.300566 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae223397-5523-4836-9d6c-7e2daf2ae313-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae223397-5523-4836-9d6c-7e2daf2ae313-plugins-config-volume\") pod \"ae223397-5523-4836-9d6c-7e2daf2ae313\" (UID: \"ae223397-5523-4836-9d6c-7e2daf2ae313\") " Oct 30 13:07:15.300730 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.300654 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4295a82c-12f5-4451-b82a-0203044f1de3-kube-api-access-wxw9q\" (UniqueName: \"kubernetes.io/projected/4295a82c-12f5-4451-b82a-0203044f1de3-kube-api-access-wxw9q\") pod \"4295a82c-12f5-4451-b82a-0203044f1de3\" (UID: \"4295a82c-12f5-4451-b82a-0203044f1de3\") " Oct 30 13:07:15.300823 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.300803 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4295a82c-12f5-4451-b82a-0203044f1de3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4295a82c-12f5-4451-b82a-0203044f1de3-plugins-config-volume\") pod \"4295a82c-12f5-4451-b82a-0203044f1de3\" (UID: \"4295a82c-12f5-4451-b82a-0203044f1de3\") " Oct 30 13:07:15.300960 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.300939 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae223397-5523-4836-9d6c-7e2daf2ae313-kube-api-access-chfnw\" (UniqueName: \"kubernetes.io/projected/ae223397-5523-4836-9d6c-7e2daf2ae313-kube-api-access-chfnw\") pod \"ae223397-5523-4836-9d6c-7e2daf2ae313\" (UID: \"ae223397-5523-4836-9d6c-7e2daf2ae313\") " Oct 30 13:07:15.301033 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.300991 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae223397-5523-4836-9d6c-7e2daf2ae313-plugins-config-volume" pod "ae223397-5523-4836-9d6c-7e2daf2ae313" (UID: "ae223397-5523-4836-9d6c-7e2daf2ae313"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:15.301244 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.301210 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4295a82c-12f5-4451-b82a-0203044f1de3-plugins-config-volume" pod "4295a82c-12f5-4451-b82a-0203044f1de3" (UID: "4295a82c-12f5-4451-b82a-0203044f1de3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:15.301647 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.301626 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae223397-5523-4836-9d6c-7e2daf2ae313-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:15.301726 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.301653 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4295a82c-12f5-4451-b82a-0203044f1de3-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:15.303061 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.303033 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4295a82c-12f5-4451-b82a-0203044f1de3-kube-api-access-wxw9q" pod "4295a82c-12f5-4451-b82a-0203044f1de3" (UID: "4295a82c-12f5-4451-b82a-0203044f1de3"). InnerVolumeSpecName "kube-api-access-wxw9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:15.303157 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.303096 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae223397-5523-4836-9d6c-7e2daf2ae313-kube-api-access-chfnw" pod "ae223397-5523-4836-9d6c-7e2daf2ae313" (UID: "ae223397-5523-4836-9d6c-7e2daf2ae313"). InnerVolumeSpecName "kube-api-access-chfnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:15.402941 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.402904 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chfnw\" (UniqueName: \"kubernetes.io/projected/ae223397-5523-4836-9d6c-7e2daf2ae313-kube-api-access-chfnw\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:15.402941 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.402935 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxw9q\" (UniqueName: \"kubernetes.io/projected/4295a82c-12f5-4451-b82a-0203044f1de3-kube-api-access-wxw9q\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:15.636756 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.636725 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:15.702890 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.702857 4413 generic.go:356] "Generic (PLEG): container finished" podID="ae223397-5523-4836-9d6c-7e2daf2ae313" containerID="6ce3aa07d7c96003104c6fff246ec2596986c6b06b7fe4afe4ce36a5abbf4d39" exitCode=0 Oct 30 13:07:15.703091 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.702911 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv" Oct 30 13:07:15.703091 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.702942 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv" event={"ID":"ae223397-5523-4836-9d6c-7e2daf2ae313","Type":"ContainerDied","Data":"6ce3aa07d7c96003104c6fff246ec2596986c6b06b7fe4afe4ce36a5abbf4d39"} Oct 30 13:07:15.703091 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.703002 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv" event={"ID":"ae223397-5523-4836-9d6c-7e2daf2ae313","Type":"ContainerDied","Data":"4376109bd5e225ee463e1132ce21163ab53c0820452ca493b7abbe287e50f604"} Oct 30 13:07:15.703091 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.703032 4413 scope.go:122] "RemoveContainer" containerID="6ce3aa07d7c96003104c6fff246ec2596986c6b06b7fe4afe4ce36a5abbf4d39" Oct 30 13:07:15.704431 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.704411 4413 generic.go:356] "Generic (PLEG): container finished" podID="4295a82c-12f5-4451-b82a-0203044f1de3" containerID="fa5c880995796b92034c6b319bc660b4b1d6bec9bc52c5e4cf5d1da08642ecc3" exitCode=0 Oct 30 13:07:15.704528 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.704478 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d" Oct 30 13:07:15.704528 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.704501 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d" event={"ID":"4295a82c-12f5-4451-b82a-0203044f1de3","Type":"ContainerDied","Data":"fa5c880995796b92034c6b319bc660b4b1d6bec9bc52c5e4cf5d1da08642ecc3"} Oct 30 13:07:15.704643 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.704576 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d" event={"ID":"4295a82c-12f5-4451-b82a-0203044f1de3","Type":"ContainerDied","Data":"a352db8ff5bf03c3b383aa546c9951a9b96abdfa439b3942e6646aa3a8a2cdcb"} Oct 30 13:07:15.715005 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.714983 4413 scope.go:122] "RemoveContainer" containerID="6ce3aa07d7c96003104c6fff246ec2596986c6b06b7fe4afe4ce36a5abbf4d39" Oct 30 13:07:15.715392 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:07:15.715366 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ce3aa07d7c96003104c6fff246ec2596986c6b06b7fe4afe4ce36a5abbf4d39\": container with ID starting with 6ce3aa07d7c96003104c6fff246ec2596986c6b06b7fe4afe4ce36a5abbf4d39 not found: ID does not exist" containerID="6ce3aa07d7c96003104c6fff246ec2596986c6b06b7fe4afe4ce36a5abbf4d39" Oct 30 13:07:15.715490 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.715399 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce3aa07d7c96003104c6fff246ec2596986c6b06b7fe4afe4ce36a5abbf4d39"} err="failed to get container status \"6ce3aa07d7c96003104c6fff246ec2596986c6b06b7fe4afe4ce36a5abbf4d39\": rpc error: code = NotFound desc = could not find container \"6ce3aa07d7c96003104c6fff246ec2596986c6b06b7fe4afe4ce36a5abbf4d39\": container with ID starting with 6ce3aa07d7c96003104c6fff246ec2596986c6b06b7fe4afe4ce36a5abbf4d39 not found: ID does not exist" Oct 30 13:07:15.715490 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.715420 4413 scope.go:122] "RemoveContainer" containerID="fa5c880995796b92034c6b319bc660b4b1d6bec9bc52c5e4cf5d1da08642ecc3" Oct 30 13:07:15.725895 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.725867 4413 scope.go:122] "RemoveContainer" containerID="fa5c880995796b92034c6b319bc660b4b1d6bec9bc52c5e4cf5d1da08642ecc3" Oct 30 13:07:15.726230 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:07:15.726210 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa5c880995796b92034c6b319bc660b4b1d6bec9bc52c5e4cf5d1da08642ecc3\": container with ID starting with fa5c880995796b92034c6b319bc660b4b1d6bec9bc52c5e4cf5d1da08642ecc3 not found: ID does not exist" containerID="fa5c880995796b92034c6b319bc660b4b1d6bec9bc52c5e4cf5d1da08642ecc3" Oct 30 13:07:15.726347 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.726239 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa5c880995796b92034c6b319bc660b4b1d6bec9bc52c5e4cf5d1da08642ecc3"} err="failed to get container status \"fa5c880995796b92034c6b319bc660b4b1d6bec9bc52c5e4cf5d1da08642ecc3\": rpc error: code = NotFound desc = could not find container \"fa5c880995796b92034c6b319bc660b4b1d6bec9bc52c5e4cf5d1da08642ecc3\": container with ID starting with fa5c880995796b92034c6b319bc660b4b1d6bec9bc52c5e4cf5d1da08642ecc3 not found: ID does not exist" Oct 30 13:07:15.754349 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.754318 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d"] Oct 30 13:07:15.760652 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.760625 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-10f364-dc7459d8d-t9d6d"] Oct 30 13:07:15.793230 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.793192 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv"] Oct 30 13:07:15.800936 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:15.800912 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-10f364-747496b57f-cfrvv"] Oct 30 13:07:17.022724 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:17.022693 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4295a82c-12f5-4451-b82a-0203044f1de3" path="/var/lib/kubelet/pods/4295a82c-12f5-4451-b82a-0203044f1de3/volumes" Oct 30 13:07:17.023165 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:17.023149 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae223397-5523-4836-9d6c-7e2daf2ae313" path="/var/lib/kubelet/pods/ae223397-5523-4836-9d6c-7e2daf2ae313/volumes" Oct 30 13:07:17.360202 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:17.360126 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:18.157321 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:18.157292 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" Oct 30 13:07:18.241119 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:18.241081 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835438be-d2d1-40da-a7ed-cd420db69319-kube-api-access-2c9k6\" (UniqueName: \"kubernetes.io/projected/835438be-d2d1-40da-a7ed-cd420db69319-kube-api-access-2c9k6\") pod \"835438be-d2d1-40da-a7ed-cd420db69319\" (UID: \"835438be-d2d1-40da-a7ed-cd420db69319\") " Oct 30 13:07:18.241306 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:18.241175 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835438be-d2d1-40da-a7ed-cd420db69319-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835438be-d2d1-40da-a7ed-cd420db69319-plugins-config-volume\") pod \"835438be-d2d1-40da-a7ed-cd420db69319\" (UID: \"835438be-d2d1-40da-a7ed-cd420db69319\") " Oct 30 13:07:18.241626 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:18.241597 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835438be-d2d1-40da-a7ed-cd420db69319-plugins-config-volume" pod "835438be-d2d1-40da-a7ed-cd420db69319" (UID: "835438be-d2d1-40da-a7ed-cd420db69319"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:18.241903 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:18.241887 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835438be-d2d1-40da-a7ed-cd420db69319-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:18.243418 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:18.243398 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835438be-d2d1-40da-a7ed-cd420db69319-kube-api-access-2c9k6" pod "835438be-d2d1-40da-a7ed-cd420db69319" (UID: "835438be-d2d1-40da-a7ed-cd420db69319"). InnerVolumeSpecName "kube-api-access-2c9k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:18.342879 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:18.342855 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c9k6\" (UniqueName: \"kubernetes.io/projected/835438be-d2d1-40da-a7ed-cd420db69319-kube-api-access-2c9k6\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:18.727960 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:18.727926 4413 generic.go:356] "Generic (PLEG): container finished" podID="835438be-d2d1-40da-a7ed-cd420db69319" containerID="f71d1c54fdd9603db20a4ee568489b69d82a344bd7de4b3051819e482d46d01c" exitCode=0 Oct 30 13:07:18.728194 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:18.728003 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" Oct 30 13:07:18.728194 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:18.728022 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" event={"ID":"835438be-d2d1-40da-a7ed-cd420db69319","Type":"ContainerDied","Data":"f71d1c54fdd9603db20a4ee568489b69d82a344bd7de4b3051819e482d46d01c"} Oct 30 13:07:18.728194 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:18.728107 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb" event={"ID":"835438be-d2d1-40da-a7ed-cd420db69319","Type":"ContainerDied","Data":"19a0faf29ccdefd783042b3c82a38290600ce699b6312790f27815a53e8822d4"} Oct 30 13:07:18.728194 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:18.728133 4413 scope.go:122] "RemoveContainer" containerID="f71d1c54fdd9603db20a4ee568489b69d82a344bd7de4b3051819e482d46d01c" Oct 30 13:07:18.738874 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:18.738852 4413 scope.go:122] "RemoveContainer" containerID="f71d1c54fdd9603db20a4ee568489b69d82a344bd7de4b3051819e482d46d01c" Oct 30 13:07:18.739229 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:07:18.739208 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71d1c54fdd9603db20a4ee568489b69d82a344bd7de4b3051819e482d46d01c\": container with ID starting with f71d1c54fdd9603db20a4ee568489b69d82a344bd7de4b3051819e482d46d01c not found: ID does not exist" containerID="f71d1c54fdd9603db20a4ee568489b69d82a344bd7de4b3051819e482d46d01c" Oct 30 13:07:18.739319 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:18.739240 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71d1c54fdd9603db20a4ee568489b69d82a344bd7de4b3051819e482d46d01c"} err="failed to get container status \"f71d1c54fdd9603db20a4ee568489b69d82a344bd7de4b3051819e482d46d01c\": rpc error: code = NotFound desc = could not find container \"f71d1c54fdd9603db20a4ee568489b69d82a344bd7de4b3051819e482d46d01c\": container with ID starting with f71d1c54fdd9603db20a4ee568489b69d82a344bd7de4b3051819e482d46d01c not found: ID does not exist" Oct 30 13:07:18.779798 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:18.779766 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb"] Oct 30 13:07:18.791618 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:18.791574 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-12fd33-584475c868-2bcrb"] Oct 30 13:07:18.960994 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:18.960964 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:19.020938 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.020864 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="835438be-d2d1-40da-a7ed-cd420db69319" path="/var/lib/kubelet/pods/835438be-d2d1-40da-a7ed-cd420db69319/volumes" Oct 30 13:07:19.433853 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.433818 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" containerID="cri-o://71dc899eeb089d4d0e80ae02460b2751bb0bdc30dc9962ab9a172dedeacf256c" gracePeriod=45 Oct 30 13:07:19.698354 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.698297 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 13:07:19.737628 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.737574 4413 generic.go:356] "Generic (PLEG): container finished" podID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerID="71dc899eeb089d4d0e80ae02460b2751bb0bdc30dc9962ab9a172dedeacf256c" exitCode=0 Oct 30 13:07:19.737865 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.737753 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" event={"ID":"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf","Type":"ContainerDied","Data":"71dc899eeb089d4d0e80ae02460b2751bb0bdc30dc9962ab9a172dedeacf256c"} Oct 30 13:07:19.737865 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.737778 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" Oct 30 13:07:19.737865 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.737797 4413 scope.go:122] "RemoveContainer" containerID="71dc899eeb089d4d0e80ae02460b2751bb0bdc30dc9962ab9a172dedeacf256c" Oct 30 13:07:19.738047 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.737786 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j" event={"ID":"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf","Type":"ContainerDied","Data":"32fe96b509bc2a09d1d343161cfdb851f8252b70a3666220e6871645e1dc398d"} Oct 30 13:07:19.749579 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.749557 4413 scope.go:122] "RemoveContainer" containerID="71dc899eeb089d4d0e80ae02460b2751bb0bdc30dc9962ab9a172dedeacf256c" Oct 30 13:07:19.749959 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:07:19.749934 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71dc899eeb089d4d0e80ae02460b2751bb0bdc30dc9962ab9a172dedeacf256c\": container with ID starting with 71dc899eeb089d4d0e80ae02460b2751bb0bdc30dc9962ab9a172dedeacf256c not found: ID does not exist" containerID="71dc899eeb089d4d0e80ae02460b2751bb0bdc30dc9962ab9a172dedeacf256c" Oct 30 13:07:19.750079 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.749965 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71dc899eeb089d4d0e80ae02460b2751bb0bdc30dc9962ab9a172dedeacf256c"} err="failed to get container status \"71dc899eeb089d4d0e80ae02460b2751bb0bdc30dc9962ab9a172dedeacf256c\": rpc error: code = NotFound desc = could not find container \"71dc899eeb089d4d0e80ae02460b2751bb0bdc30dc9962ab9a172dedeacf256c\": container with ID starting with 71dc899eeb089d4d0e80ae02460b2751bb0bdc30dc9962ab9a172dedeacf256c not found: ID does not exist" Oct 30 13:07:19.759313 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.759282 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-kube-api-access-lxv8t\" (UniqueName: \"kubernetes.io/projected/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-kube-api-access-lxv8t\") pod \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " Oct 30 13:07:19.759491 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.759363 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-tls-certs\") pod \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " Oct 30 13:07:19.759491 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.759457 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-model-cache\") pod \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " Oct 30 13:07:19.759621 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.759526 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-home\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-home\") pod \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " Oct 30 13:07:19.759687 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.759629 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-dshm\") pod \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " Oct 30 13:07:19.759751 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.759686 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-tmp-dir\") pod \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\" (UID: \"93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf\") " Oct 30 13:07:19.760030 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.760004 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-home" pod "93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" (UID: "93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.760294 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.760212 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-model-cache" pod "93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" (UID: "93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.760402 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.760300 4413 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.760635 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.760529 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-tmp-dir" pod "93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" (UID: "93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.762241 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.762219 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-dshm" pod "93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" (UID: "93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:19.762441 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.762425 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-tls-certs" pod "93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" (UID: "93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:19.766388 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.766366 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-kube-api-access-lxv8t" pod "93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" (UID: "93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf"). InnerVolumeSpecName "kube-api-access-lxv8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:19.861604 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.861572 4413 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.861604 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.861602 4413 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.861807 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.861622 4413 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.861807 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.861631 4413 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:19.861807 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:19.861640 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxv8t\" (UniqueName: \"kubernetes.io/projected/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf-kube-api-access-lxv8t\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:20.005735 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:20.005654 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj"] Oct 30 13:07:20.042124 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:20.042087 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl"] Oct 30 13:07:20.134779 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:20.134748 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j"] Oct 30 13:07:20.138008 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:20.137933 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f920f7/distinct-w0-f920f7-kserve-6c667ddc4f-qp57j"] Oct 30 13:07:20.165208 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:20.165176 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:20.288940 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:20.288860 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:20.684880 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:20.684849 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:21.020787 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:21.020715 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" path="/var/lib/kubelet/pods/93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf/volumes" Oct 30 13:07:22.264745 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:22.264718 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:23.861767 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:23.861732 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:25.006930 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.006879 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" podUID="0818b575-9f29-4241-967b-448947aaf982" containerName="payload-processing" containerID="cri-o://a58fa4287c50b9b20a822260f3e9dd75c85630bc58d1298d813a70f9ba900471" gracePeriod=25 Oct 30 13:07:25.042555 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.042521 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" podUID="e2971b88-5b87-49db-b7f8-60c2a9d7116d" containerName="payload-pre-processing" containerID="cri-o://27f9a0f03583d6206ee5731025fcc540132eb075f406148a0b57c4860f39036d" gracePeriod=25 Oct 30 13:07:25.132619 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.132595 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b"] Oct 30 13:07:25.163537 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.163510 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2"] Oct 30 13:07:25.234849 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.234817 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs"] Oct 30 13:07:25.235145 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.235118 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" podUID="d94ea46b-0895-4536-a9c6-4088e1116400" containerName="maas-api" containerID="cri-o://28ae203f5d4784887f343bf9d9d2ea4abc3cab289b2301cdb6351d7ce03ad03e" gracePeriod=30 Oct 30 13:07:25.292608 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.292565 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:25.412765 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.412727 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:25.499755 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.499731 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" Oct 30 13:07:25.610685 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.610606 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn"] Oct 30 13:07:25.610910 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.610882 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" podUID="5bfd491e-874f-4edd-b01b-b5bc8d20278b" containerName="maas-api" containerID="cri-o://a2500660e94b67ad0df3124833da41e9cdaa7d4cb9925d267b737a85739f8793" gracePeriod=30 Oct 30 13:07:25.631480 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.631453 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d94ea46b-0895-4536-a9c6-4088e1116400-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d94ea46b-0895-4536-a9c6-4088e1116400-maas-api-tls\") pod \"d94ea46b-0895-4536-a9c6-4088e1116400\" (UID: \"d94ea46b-0895-4536-a9c6-4088e1116400\") " Oct 30 13:07:25.631615 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.631549 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d94ea46b-0895-4536-a9c6-4088e1116400-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94ea46b-0895-4536-a9c6-4088e1116400-openshift-service-ca\") pod \"d94ea46b-0895-4536-a9c6-4088e1116400\" (UID: \"d94ea46b-0895-4536-a9c6-4088e1116400\") " Oct 30 13:07:25.631712 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.631694 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d94ea46b-0895-4536-a9c6-4088e1116400-kube-api-access-2mmpf\" (UniqueName: \"kubernetes.io/projected/d94ea46b-0895-4536-a9c6-4088e1116400-kube-api-access-2mmpf\") pod \"d94ea46b-0895-4536-a9c6-4088e1116400\" (UID: \"d94ea46b-0895-4536-a9c6-4088e1116400\") " Oct 30 13:07:25.631856 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.631838 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d94ea46b-0895-4536-a9c6-4088e1116400-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d94ea46b-0895-4536-a9c6-4088e1116400-metrics-certs\") pod \"d94ea46b-0895-4536-a9c6-4088e1116400\" (UID: \"d94ea46b-0895-4536-a9c6-4088e1116400\") " Oct 30 13:07:25.632442 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.632413 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d94ea46b-0895-4536-a9c6-4088e1116400-openshift-service-ca" pod "d94ea46b-0895-4536-a9c6-4088e1116400" (UID: "d94ea46b-0895-4536-a9c6-4088e1116400"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:25.634955 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.634885 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d94ea46b-0895-4536-a9c6-4088e1116400-kube-api-access-2mmpf" pod "d94ea46b-0895-4536-a9c6-4088e1116400" (UID: "d94ea46b-0895-4536-a9c6-4088e1116400"). InnerVolumeSpecName "kube-api-access-2mmpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:25.635363 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.635329 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d94ea46b-0895-4536-a9c6-4088e1116400-maas-api-tls" pod "d94ea46b-0895-4536-a9c6-4088e1116400" (UID: "d94ea46b-0895-4536-a9c6-4088e1116400"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:25.635363 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.635342 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d94ea46b-0895-4536-a9c6-4088e1116400-metrics-certs" pod "d94ea46b-0895-4536-a9c6-4088e1116400" (UID: "d94ea46b-0895-4536-a9c6-4088e1116400"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:25.687383 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.687353 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:25.734086 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.734055 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d94ea46b-0895-4536-a9c6-4088e1116400-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:25.734180 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.734092 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d94ea46b-0895-4536-a9c6-4088e1116400-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:25.734180 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.734107 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94ea46b-0895-4536-a9c6-4088e1116400-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:25.734180 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.734123 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mmpf\" (UniqueName: \"kubernetes.io/projected/d94ea46b-0895-4536-a9c6-4088e1116400-kube-api-access-2mmpf\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:25.793166 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.793138 4413 generic.go:356] "Generic (PLEG): container finished" podID="5bfd491e-874f-4edd-b01b-b5bc8d20278b" containerID="a2500660e94b67ad0df3124833da41e9cdaa7d4cb9925d267b737a85739f8793" exitCode=0 Oct 30 13:07:25.793345 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.793218 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" event={"ID":"5bfd491e-874f-4edd-b01b-b5bc8d20278b","Type":"ContainerDied","Data":"a2500660e94b67ad0df3124833da41e9cdaa7d4cb9925d267b737a85739f8793"} Oct 30 13:07:25.794925 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.794890 4413 generic.go:356] "Generic (PLEG): container finished" podID="d94ea46b-0895-4536-a9c6-4088e1116400" containerID="28ae203f5d4784887f343bf9d9d2ea4abc3cab289b2301cdb6351d7ce03ad03e" exitCode=0 Oct 30 13:07:25.795034 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.794946 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" Oct 30 13:07:25.795034 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.794944 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" event={"ID":"d94ea46b-0895-4536-a9c6-4088e1116400","Type":"ContainerDied","Data":"28ae203f5d4784887f343bf9d9d2ea4abc3cab289b2301cdb6351d7ce03ad03e"} Oct 30 13:07:25.795164 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.795054 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs" event={"ID":"d94ea46b-0895-4536-a9c6-4088e1116400","Type":"ContainerDied","Data":"bb4ed41ce38e01cf2193ea9cea119659d736411d1782b9800c018be7b73bc4db"} Oct 30 13:07:25.795164 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.795110 4413 scope.go:122] "RemoveContainer" containerID="28ae203f5d4784887f343bf9d9d2ea4abc3cab289b2301cdb6351d7ce03ad03e" Oct 30 13:07:25.805279 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.805232 4413 scope.go:122] "RemoveContainer" containerID="28ae203f5d4784887f343bf9d9d2ea4abc3cab289b2301cdb6351d7ce03ad03e" Oct 30 13:07:25.805722 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:07:25.805592 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ae203f5d4784887f343bf9d9d2ea4abc3cab289b2301cdb6351d7ce03ad03e\": container with ID starting with 28ae203f5d4784887f343bf9d9d2ea4abc3cab289b2301cdb6351d7ce03ad03e not found: ID does not exist" containerID="28ae203f5d4784887f343bf9d9d2ea4abc3cab289b2301cdb6351d7ce03ad03e" Oct 30 13:07:25.805722 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.805632 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ae203f5d4784887f343bf9d9d2ea4abc3cab289b2301cdb6351d7ce03ad03e"} err="failed to get container status \"28ae203f5d4784887f343bf9d9d2ea4abc3cab289b2301cdb6351d7ce03ad03e\": rpc error: code = NotFound desc = could not find container \"28ae203f5d4784887f343bf9d9d2ea4abc3cab289b2301cdb6351d7ce03ad03e\": container with ID starting with 28ae203f5d4784887f343bf9d9d2ea4abc3cab289b2301cdb6351d7ce03ad03e not found: ID does not exist" Oct 30 13:07:25.877399 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.872070 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs"] Oct 30 13:07:25.877399 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.876493 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2675c0-88dd5ccb5-mdzrs"] Oct 30 13:07:25.901748 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.901707 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" Oct 30 13:07:25.997039 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.997009 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l"] Oct 30 13:07:25.997380 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:25.997354 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" podUID="4aa9879d-dab4-42ea-8dd8-ad6bba64214c" containerName="maas-api" containerID="cri-o://7c7d28c6baf25f3824751a21c640b20082a55a88b1aa783269d24a70f16deaff" gracePeriod=30 Oct 30 13:07:26.037396 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.037373 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfd491e-874f-4edd-b01b-b5bc8d20278b-kube-api-access-jgrjj\" (UniqueName: \"kubernetes.io/projected/5bfd491e-874f-4edd-b01b-b5bc8d20278b-kube-api-access-jgrjj\") pod \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\" (UID: \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\") " Oct 30 13:07:26.037837 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.037491 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bfd491e-874f-4edd-b01b-b5bc8d20278b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bfd491e-874f-4edd-b01b-b5bc8d20278b-maas-api-tls\") pod \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\" (UID: \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\") " Oct 30 13:07:26.037837 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.037516 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bfd491e-874f-4edd-b01b-b5bc8d20278b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bfd491e-874f-4edd-b01b-b5bc8d20278b-metrics-certs\") pod \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\" (UID: \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\") " Oct 30 13:07:26.037837 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.037717 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bfd491e-874f-4edd-b01b-b5bc8d20278b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bfd491e-874f-4edd-b01b-b5bc8d20278b-openshift-service-ca\") pod \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\" (UID: \"5bfd491e-874f-4edd-b01b-b5bc8d20278b\") " Oct 30 13:07:26.038318 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.038282 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfd491e-874f-4edd-b01b-b5bc8d20278b-openshift-service-ca" pod "5bfd491e-874f-4edd-b01b-b5bc8d20278b" (UID: "5bfd491e-874f-4edd-b01b-b5bc8d20278b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:26.040426 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.040399 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bfd491e-874f-4edd-b01b-b5bc8d20278b-maas-api-tls" pod "5bfd491e-874f-4edd-b01b-b5bc8d20278b" (UID: "5bfd491e-874f-4edd-b01b-b5bc8d20278b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:26.040597 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.040577 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bfd491e-874f-4edd-b01b-b5bc8d20278b-metrics-certs" pod "5bfd491e-874f-4edd-b01b-b5bc8d20278b" (UID: "5bfd491e-874f-4edd-b01b-b5bc8d20278b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:26.045220 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.045197 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfd491e-874f-4edd-b01b-b5bc8d20278b-kube-api-access-jgrjj" pod "5bfd491e-874f-4edd-b01b-b5bc8d20278b" (UID: "5bfd491e-874f-4edd-b01b-b5bc8d20278b"). InnerVolumeSpecName "kube-api-access-jgrjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:26.070592 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.070571 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:26.139919 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.139867 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bfd491e-874f-4edd-b01b-b5bc8d20278b-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:26.139919 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.139891 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bfd491e-874f-4edd-b01b-b5bc8d20278b-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:26.139919 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.139900 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bfd491e-874f-4edd-b01b-b5bc8d20278b-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:26.139919 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.139909 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgrjj\" (UniqueName: \"kubernetes.io/projected/5bfd491e-874f-4edd-b01b-b5bc8d20278b-kube-api-access-jgrjj\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:26.184011 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.183985 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:26.246684 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.246662 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" Oct 30 13:07:26.341540 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.341510 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-kube-api-access-d6p7d\" (UniqueName: \"kubernetes.io/projected/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-kube-api-access-d6p7d\") pod \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\" (UID: \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\") " Oct 30 13:07:26.341697 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.341575 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-maas-api-tls\") pod \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\" (UID: \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\") " Oct 30 13:07:26.341697 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.341671 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-openshift-service-ca\") pod \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\" (UID: \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\") " Oct 30 13:07:26.341842 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.341767 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-metrics-certs\") pod \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\" (UID: \"4aa9879d-dab4-42ea-8dd8-ad6bba64214c\") " Oct 30 13:07:26.342060 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.342025 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-openshift-service-ca" pod "4aa9879d-dab4-42ea-8dd8-ad6bba64214c" (UID: "4aa9879d-dab4-42ea-8dd8-ad6bba64214c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:26.342501 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.342482 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:26.343884 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.343853 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-kube-api-access-d6p7d" pod "4aa9879d-dab4-42ea-8dd8-ad6bba64214c" (UID: "4aa9879d-dab4-42ea-8dd8-ad6bba64214c"). InnerVolumeSpecName "kube-api-access-d6p7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:26.343978 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.343925 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-maas-api-tls" pod "4aa9879d-dab4-42ea-8dd8-ad6bba64214c" (UID: "4aa9879d-dab4-42ea-8dd8-ad6bba64214c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:26.343978 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.343942 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-metrics-certs" pod "4aa9879d-dab4-42ea-8dd8-ad6bba64214c" (UID: "4aa9879d-dab4-42ea-8dd8-ad6bba64214c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:26.387302 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.387280 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:26.453366 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.445204 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6p7d\" (UniqueName: \"kubernetes.io/projected/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-kube-api-access-d6p7d\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:26.453366 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.445265 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:26.453366 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.445280 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa9879d-dab4-42ea-8dd8-ad6bba64214c-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:26.685810 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.685777 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:26.802938 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.802843 4413 generic.go:356] "Generic (PLEG): container finished" podID="4aa9879d-dab4-42ea-8dd8-ad6bba64214c" containerID="7c7d28c6baf25f3824751a21c640b20082a55a88b1aa783269d24a70f16deaff" exitCode=0 Oct 30 13:07:26.802938 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.802893 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" Oct 30 13:07:26.803188 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.802929 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" event={"ID":"4aa9879d-dab4-42ea-8dd8-ad6bba64214c","Type":"ContainerDied","Data":"7c7d28c6baf25f3824751a21c640b20082a55a88b1aa783269d24a70f16deaff"} Oct 30 13:07:26.803188 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.802989 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l" event={"ID":"4aa9879d-dab4-42ea-8dd8-ad6bba64214c","Type":"ContainerDied","Data":"5e7c7715eb4809401562e37158d69ba0ff878a6ff768073e001d2ba144d045c2"} Oct 30 13:07:26.803188 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.803015 4413 scope.go:122] "RemoveContainer" containerID="7c7d28c6baf25f3824751a21c640b20082a55a88b1aa783269d24a70f16deaff" Oct 30 13:07:26.815034 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.814751 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" event={"ID":"5bfd491e-874f-4edd-b01b-b5bc8d20278b","Type":"ContainerDied","Data":"71b27408ce37910252991bee6078a6e9780ea5504b42740246ac76241a098081"} Oct 30 13:07:26.815034 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.814972 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn" Oct 30 13:07:26.834420 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.834351 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:26.837846 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.837814 4413 scope.go:122] "RemoveContainer" containerID="7c7d28c6baf25f3824751a21c640b20082a55a88b1aa783269d24a70f16deaff" Oct 30 13:07:26.838225 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:07:26.838199 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c7d28c6baf25f3824751a21c640b20082a55a88b1aa783269d24a70f16deaff\": container with ID starting with 7c7d28c6baf25f3824751a21c640b20082a55a88b1aa783269d24a70f16deaff not found: ID does not exist" containerID="7c7d28c6baf25f3824751a21c640b20082a55a88b1aa783269d24a70f16deaff" Oct 30 13:07:26.838356 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.838231 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c7d28c6baf25f3824751a21c640b20082a55a88b1aa783269d24a70f16deaff"} err="failed to get container status \"7c7d28c6baf25f3824751a21c640b20082a55a88b1aa783269d24a70f16deaff\": rpc error: code = NotFound desc = could not find container \"7c7d28c6baf25f3824751a21c640b20082a55a88b1aa783269d24a70f16deaff\": container with ID starting with 7c7d28c6baf25f3824751a21c640b20082a55a88b1aa783269d24a70f16deaff not found: ID does not exist" Oct 30 13:07:26.838356 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.838270 4413 scope.go:122] "RemoveContainer" containerID="a2500660e94b67ad0df3124833da41e9cdaa7d4cb9925d267b737a85739f8793" Oct 30 13:07:26.879191 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.879161 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l"] Oct 30 13:07:26.884238 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.884217 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-10f364-5b5d6796d7-pgg8l"] Oct 30 13:07:26.917736 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.917716 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn"] Oct 30 13:07:26.924419 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:26.924400 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-812990-6679bf55b4-4hvzn"] Oct 30 13:07:27.021965 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:27.021929 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa9879d-dab4-42ea-8dd8-ad6bba64214c" path="/var/lib/kubelet/pods/4aa9879d-dab4-42ea-8dd8-ad6bba64214c/volumes" Oct 30 13:07:27.022457 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:27.022437 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bfd491e-874f-4edd-b01b-b5bc8d20278b" path="/var/lib/kubelet/pods/5bfd491e-874f-4edd-b01b-b5bc8d20278b/volumes" Oct 30 13:07:27.022912 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:27.022893 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d94ea46b-0895-4536-a9c6-4088e1116400" path="/var/lib/kubelet/pods/d94ea46b-0895-4536-a9c6-4088e1116400/volumes" Oct 30 13:07:27.054537 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:27.054479 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:27.275601 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:27.275566 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:27.401835 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:27.401807 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:27.709423 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:27.709310 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:28.768999 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:28.768967 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:30.133849 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:30.133795 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" podUID="68a11193-ed13-452f-b0b1-eaeb60b21465" containerName="payload-processing" containerID="cri-o://46bc5abd3a94b3010a0f89c1cf734cadbd3fe16580474ee674b503025f9e3226" gracePeriod=25 Oct 30 13:07:30.164293 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:30.164226 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" podUID="fdabf3ad-4ef3-4f8e-9f53-6565dacf505e" containerName="payload-pre-processing" containerID="cri-o://baccd1ee02487a3f1a999ea59dfe371f52d2dfa486530e92d16cd2d3fd6f869d" gracePeriod=25 Oct 30 13:07:30.458970 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:30.458901 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:30.663441 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:30.663400 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" podUID="68a11193-ed13-452f-b0b1-eaeb60b21465" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:31.670837 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:31.670795 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" podUID="fdabf3ad-4ef3-4f8e-9f53-6565dacf505e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:31.671238 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:31.670879 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" podUID="0818b575-9f29-4241-967b-448947aaf982" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:32.050447 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:32.050367 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:32.681016 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:32.680979 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" podUID="e2971b88-5b87-49db-b7f8-60c2a9d7116d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:33.729862 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:33.729787 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:33.978018 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:33.977983 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:34.108195 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:34.108109 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:34.294201 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:34.294162 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq"] Oct 30 13:07:34.294480 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:34.294455 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" podUID="5b66ed77-d0a7-4180-ba18-8598280715af" containerName="istio-proxy" containerID="cri-o://09521b004b5631deadd8b0939e59fe9fa8dc0e65b25934db4edbeb48869526a4" gracePeriod=30 Oct 30 13:07:34.326675 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:34.326630 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:34.561824 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:34.561791 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:34.688851 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:34.688816 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:34.909317 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:34.909285 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:35.649583 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:35.649550 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:35.785697 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:35.785667 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:35.813072 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:07:35.813041 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:07:35.910402 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:35.910319 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:36.125189 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:36.125157 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2"] Oct 30 13:07:36.125468 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:36.125441 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" podUID="75e43f0b-df10-4b4e-b7af-67d515178a84" containerName="istio-proxy" containerID="cri-o://e6f0e6d5056b87faa76e90092b3215e9f9c06fa47b9a12684da57d8840c9501c" gracePeriod=30 Oct 30 13:07:36.145223 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:36.145200 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:37.571035 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:37.571003 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:39.261184 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.261151 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:39.560801 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.560778 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 13:07:39.599592 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.599568 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b66ed77-d0a7-4180-ba18-8598280715af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b66ed77-d0a7-4180-ba18-8598280715af-istiod-ca-cert\") pod \"5b66ed77-d0a7-4180-ba18-8598280715af\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " Oct 30 13:07:39.599724 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.599617 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-istio-envoy\") pod \"5b66ed77-d0a7-4180-ba18-8598280715af\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " Oct 30 13:07:39.599724 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.599637 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-workload-socket\") pod \"5b66ed77-d0a7-4180-ba18-8598280715af\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " Oct 30 13:07:39.599724 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.599717 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b66ed77-d0a7-4180-ba18-8598280715af-istio-token\" (UniqueName: \"kubernetes.io/projected/5b66ed77-d0a7-4180-ba18-8598280715af-istio-token\") pod \"5b66ed77-d0a7-4180-ba18-8598280715af\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " Oct 30 13:07:39.599912 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.599741 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b66ed77-d0a7-4180-ba18-8598280715af-kube-api-access-dsgzm\" (UniqueName: \"kubernetes.io/projected/5b66ed77-d0a7-4180-ba18-8598280715af-kube-api-access-dsgzm\") pod \"5b66ed77-d0a7-4180-ba18-8598280715af\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " Oct 30 13:07:39.599912 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.599758 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-istio-data\") pod \"5b66ed77-d0a7-4180-ba18-8598280715af\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " Oct 30 13:07:39.599912 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.599837 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-credential-socket\") pod \"5b66ed77-d0a7-4180-ba18-8598280715af\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " Oct 30 13:07:39.599912 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.599874 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b66ed77-d0a7-4180-ba18-8598280715af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b66ed77-d0a7-4180-ba18-8598280715af-istio-podinfo\") pod \"5b66ed77-d0a7-4180-ba18-8598280715af\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " Oct 30 13:07:39.600140 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.599919 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-workload-certs\") pod \"5b66ed77-d0a7-4180-ba18-8598280715af\" (UID: \"5b66ed77-d0a7-4180-ba18-8598280715af\") " Oct 30 13:07:39.600140 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.599959 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b66ed77-d0a7-4180-ba18-8598280715af-istiod-ca-cert" pod "5b66ed77-d0a7-4180-ba18-8598280715af" (UID: "5b66ed77-d0a7-4180-ba18-8598280715af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:39.600293 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.600216 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-istio-data" pod "5b66ed77-d0a7-4180-ba18-8598280715af" (UID: "5b66ed77-d0a7-4180-ba18-8598280715af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:39.600391 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.600307 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-credential-socket" pod "5b66ed77-d0a7-4180-ba18-8598280715af" (UID: "5b66ed77-d0a7-4180-ba18-8598280715af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:39.600601 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.600458 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-workload-certs" pod "5b66ed77-d0a7-4180-ba18-8598280715af" (UID: "5b66ed77-d0a7-4180-ba18-8598280715af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:39.600601 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.600514 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:39.600601 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.600534 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b66ed77-d0a7-4180-ba18-8598280715af-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:39.600601 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.600549 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:39.600601 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.600572 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-workload-socket" pod "5b66ed77-d0a7-4180-ba18-8598280715af" (UID: "5b66ed77-d0a7-4180-ba18-8598280715af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:39.602004 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.601980 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b66ed77-d0a7-4180-ba18-8598280715af-kube-api-access-dsgzm" pod "5b66ed77-d0a7-4180-ba18-8598280715af" (UID: "5b66ed77-d0a7-4180-ba18-8598280715af"). InnerVolumeSpecName "kube-api-access-dsgzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:39.602142 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.602088 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b66ed77-d0a7-4180-ba18-8598280715af-istio-token" pod "5b66ed77-d0a7-4180-ba18-8598280715af" (UID: "5b66ed77-d0a7-4180-ba18-8598280715af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:39.602142 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.602131 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-istio-envoy" pod "5b66ed77-d0a7-4180-ba18-8598280715af" (UID: "5b66ed77-d0a7-4180-ba18-8598280715af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:39.602381 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.602365 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b66ed77-d0a7-4180-ba18-8598280715af-istio-podinfo" pod "5b66ed77-d0a7-4180-ba18-8598280715af" (UID: "5b66ed77-d0a7-4180-ba18-8598280715af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:07:39.701316 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.701275 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b66ed77-d0a7-4180-ba18-8598280715af-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:39.701316 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.701316 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsgzm\" (UniqueName: \"kubernetes.io/projected/5b66ed77-d0a7-4180-ba18-8598280715af-kube-api-access-dsgzm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:39.701316 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.701326 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b66ed77-d0a7-4180-ba18-8598280715af-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:39.701546 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.701335 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:39.701546 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.701343 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:39.701546 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.701350 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66ed77-d0a7-4180-ba18-8598280715af-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:39.901423 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.901386 4413 generic.go:356] "Generic (PLEG): container finished" podID="5b66ed77-d0a7-4180-ba18-8598280715af" containerID="09521b004b5631deadd8b0939e59fe9fa8dc0e65b25934db4edbeb48869526a4" exitCode=0 Oct 30 13:07:39.901600 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.901439 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" Oct 30 13:07:39.901600 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.901467 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" event={"ID":"5b66ed77-d0a7-4180-ba18-8598280715af","Type":"ContainerDied","Data":"09521b004b5631deadd8b0939e59fe9fa8dc0e65b25934db4edbeb48869526a4"} Oct 30 13:07:39.901600 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.901525 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq" event={"ID":"5b66ed77-d0a7-4180-ba18-8598280715af","Type":"ContainerDied","Data":"23c8a49e2dbf1f5c73e1bd1d17a02f543ce4e06254715c9bf52aaa857b56718f"} Oct 30 13:07:39.901600 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.901553 4413 scope.go:122] "RemoveContainer" containerID="09521b004b5631deadd8b0939e59fe9fa8dc0e65b25934db4edbeb48869526a4" Oct 30 13:07:39.912066 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.912048 4413 scope.go:122] "RemoveContainer" containerID="d5eb27e80038c448c69324e58fde9eb52845e69e86953a36445d70c517601905" Oct 30 13:07:39.921327 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.921309 4413 scope.go:122] "RemoveContainer" containerID="09521b004b5631deadd8b0939e59fe9fa8dc0e65b25934db4edbeb48869526a4" Oct 30 13:07:39.921591 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:07:39.921576 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09521b004b5631deadd8b0939e59fe9fa8dc0e65b25934db4edbeb48869526a4\": container with ID starting with 09521b004b5631deadd8b0939e59fe9fa8dc0e65b25934db4edbeb48869526a4 not found: ID does not exist" containerID="09521b004b5631deadd8b0939e59fe9fa8dc0e65b25934db4edbeb48869526a4" Oct 30 13:07:39.921667 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.921598 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09521b004b5631deadd8b0939e59fe9fa8dc0e65b25934db4edbeb48869526a4"} err="failed to get container status \"09521b004b5631deadd8b0939e59fe9fa8dc0e65b25934db4edbeb48869526a4\": rpc error: code = NotFound desc = could not find container \"09521b004b5631deadd8b0939e59fe9fa8dc0e65b25934db4edbeb48869526a4\": container with ID starting with 09521b004b5631deadd8b0939e59fe9fa8dc0e65b25934db4edbeb48869526a4 not found: ID does not exist" Oct 30 13:07:39.921667 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.921623 4413 scope.go:122] "RemoveContainer" containerID="d5eb27e80038c448c69324e58fde9eb52845e69e86953a36445d70c517601905" Oct 30 13:07:39.921850 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:07:39.921831 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5eb27e80038c448c69324e58fde9eb52845e69e86953a36445d70c517601905\": container with ID starting with d5eb27e80038c448c69324e58fde9eb52845e69e86953a36445d70c517601905 not found: ID does not exist" containerID="d5eb27e80038c448c69324e58fde9eb52845e69e86953a36445d70c517601905" Oct 30 13:07:39.921895 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.921856 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5eb27e80038c448c69324e58fde9eb52845e69e86953a36445d70c517601905"} err="failed to get container status \"d5eb27e80038c448c69324e58fde9eb52845e69e86953a36445d70c517601905\": rpc error: code = NotFound desc = could not find container \"d5eb27e80038c448c69324e58fde9eb52845e69e86953a36445d70c517601905\": container with ID starting with d5eb27e80038c448c69324e58fde9eb52845e69e86953a36445d70c517601905 not found: ID does not exist" Oct 30 13:07:39.958425 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.958400 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq"] Oct 30 13:07:39.970228 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:39.970206 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-812990-openshift-default-875ccc45-lfjcq"] Oct 30 13:07:40.662886 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:40.662841 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" podUID="68a11193-ed13-452f-b0b1-eaeb60b21465" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:40.983354 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:40.983276 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:41.021062 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.021026 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b66ed77-d0a7-4180-ba18-8598280715af" path="/var/lib/kubelet/pods/5b66ed77-d0a7-4180-ba18-8598280715af/volumes" Oct 30 13:07:41.392693 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.392673 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 13:07:41.417913 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.417885 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-workload-certs\") pod \"75e43f0b-df10-4b4e-b7af-67d515178a84\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " Oct 30 13:07:41.418050 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.417931 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-data\") pod \"75e43f0b-df10-4b4e-b7af-67d515178a84\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " Oct 30 13:07:41.418050 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.417999 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e43f0b-df10-4b4e-b7af-67d515178a84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75e43f0b-df10-4b4e-b7af-67d515178a84-istiod-ca-cert\") pod \"75e43f0b-df10-4b4e-b7af-67d515178a84\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " Oct 30 13:07:41.418168 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.418061 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-podinfo\") pod \"75e43f0b-df10-4b4e-b7af-67d515178a84\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " Oct 30 13:07:41.418168 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.418098 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-envoy\") pod \"75e43f0b-df10-4b4e-b7af-67d515178a84\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " Oct 30 13:07:41.418168 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.418150 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-workload-certs" pod "75e43f0b-df10-4b4e-b7af-67d515178a84" (UID: "75e43f0b-df10-4b4e-b7af-67d515178a84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:41.418374 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.418172 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-workload-socket\") pod \"75e43f0b-df10-4b4e-b7af-67d515178a84\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " Oct 30 13:07:41.418374 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.418188 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-data" pod "75e43f0b-df10-4b4e-b7af-67d515178a84" (UID: "75e43f0b-df10-4b4e-b7af-67d515178a84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:41.418374 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.418205 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-token\" (UniqueName: \"kubernetes.io/projected/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-token\") pod \"75e43f0b-df10-4b4e-b7af-67d515178a84\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " Oct 30 13:07:41.418374 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.418321 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-credential-socket\") pod \"75e43f0b-df10-4b4e-b7af-67d515178a84\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " Oct 30 13:07:41.418587 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.418377 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e43f0b-df10-4b4e-b7af-67d515178a84-kube-api-access-z7sgl\" (UniqueName: \"kubernetes.io/projected/75e43f0b-df10-4b4e-b7af-67d515178a84-kube-api-access-z7sgl\") pod \"75e43f0b-df10-4b4e-b7af-67d515178a84\" (UID: \"75e43f0b-df10-4b4e-b7af-67d515178a84\") " Oct 30 13:07:41.418587 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.418434 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-workload-socket" pod "75e43f0b-df10-4b4e-b7af-67d515178a84" (UID: "75e43f0b-df10-4b4e-b7af-67d515178a84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:41.418587 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.418460 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e43f0b-df10-4b4e-b7af-67d515178a84-istiod-ca-cert" pod "75e43f0b-df10-4b4e-b7af-67d515178a84" (UID: "75e43f0b-df10-4b4e-b7af-67d515178a84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:41.418724 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.418600 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-credential-socket" pod "75e43f0b-df10-4b4e-b7af-67d515178a84" (UID: "75e43f0b-df10-4b4e-b7af-67d515178a84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:41.418884 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.418857 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:41.418884 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.418877 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:41.418967 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.418889 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75e43f0b-df10-4b4e-b7af-67d515178a84-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:41.418967 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.418897 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:41.418967 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.418905 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:41.420436 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.420412 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-envoy" pod "75e43f0b-df10-4b4e-b7af-67d515178a84" (UID: "75e43f0b-df10-4b4e-b7af-67d515178a84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:07:41.420907 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.420880 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e43f0b-df10-4b4e-b7af-67d515178a84-kube-api-access-z7sgl" pod "75e43f0b-df10-4b4e-b7af-67d515178a84" (UID: "75e43f0b-df10-4b4e-b7af-67d515178a84"). InnerVolumeSpecName "kube-api-access-z7sgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:41.420971 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.420935 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-podinfo" pod "75e43f0b-df10-4b4e-b7af-67d515178a84" (UID: "75e43f0b-df10-4b4e-b7af-67d515178a84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:07:41.421017 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.420965 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-token" pod "75e43f0b-df10-4b4e-b7af-67d515178a84" (UID: "75e43f0b-df10-4b4e-b7af-67d515178a84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:41.520518 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.520444 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:41.520518 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.520469 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:41.520518 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.520477 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75e43f0b-df10-4b4e-b7af-67d515178a84-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:41.520518 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.520485 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7sgl\" (UniqueName: \"kubernetes.io/projected/75e43f0b-df10-4b4e-b7af-67d515178a84-kube-api-access-z7sgl\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:41.671474 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.671438 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" podUID="0818b575-9f29-4241-967b-448947aaf982" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:41.671474 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.671469 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" podUID="fdabf3ad-4ef3-4f8e-9f53-6565dacf505e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:41.917800 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.917767 4413 generic.go:356] "Generic (PLEG): container finished" podID="75e43f0b-df10-4b4e-b7af-67d515178a84" containerID="e6f0e6d5056b87faa76e90092b3215e9f9c06fa47b9a12684da57d8840c9501c" exitCode=0 Oct 30 13:07:41.917994 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.917820 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" Oct 30 13:07:41.917994 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.917844 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" event={"ID":"75e43f0b-df10-4b4e-b7af-67d515178a84","Type":"ContainerDied","Data":"e6f0e6d5056b87faa76e90092b3215e9f9c06fa47b9a12684da57d8840c9501c"} Oct 30 13:07:41.917994 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.917900 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2" event={"ID":"75e43f0b-df10-4b4e-b7af-67d515178a84","Type":"ContainerDied","Data":"806529ba39b622e2cac8a837ced95f7c69b52c66a61513f9f32e32b47aff4707"} Oct 30 13:07:41.917994 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.917921 4413 scope.go:122] "RemoveContainer" containerID="e6f0e6d5056b87faa76e90092b3215e9f9c06fa47b9a12684da57d8840c9501c" Oct 30 13:07:41.928549 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.928533 4413 scope.go:122] "RemoveContainer" containerID="e1f4a8c880313b406b4cd2b36e5344c75bdc8e7408a619bb8f538930f572d732" Oct 30 13:07:41.937631 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.937594 4413 scope.go:122] "RemoveContainer" containerID="e6f0e6d5056b87faa76e90092b3215e9f9c06fa47b9a12684da57d8840c9501c" Oct 30 13:07:41.937893 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:07:41.937875 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f0e6d5056b87faa76e90092b3215e9f9c06fa47b9a12684da57d8840c9501c\": container with ID starting with e6f0e6d5056b87faa76e90092b3215e9f9c06fa47b9a12684da57d8840c9501c not found: ID does not exist" containerID="e6f0e6d5056b87faa76e90092b3215e9f9c06fa47b9a12684da57d8840c9501c" Oct 30 13:07:41.937946 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.937900 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f0e6d5056b87faa76e90092b3215e9f9c06fa47b9a12684da57d8840c9501c"} err="failed to get container status \"e6f0e6d5056b87faa76e90092b3215e9f9c06fa47b9a12684da57d8840c9501c\": rpc error: code = NotFound desc = could not find container \"e6f0e6d5056b87faa76e90092b3215e9f9c06fa47b9a12684da57d8840c9501c\": container with ID starting with e6f0e6d5056b87faa76e90092b3215e9f9c06fa47b9a12684da57d8840c9501c not found: ID does not exist" Oct 30 13:07:41.937946 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.937917 4413 scope.go:122] "RemoveContainer" containerID="e1f4a8c880313b406b4cd2b36e5344c75bdc8e7408a619bb8f538930f572d732" Oct 30 13:07:41.938139 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:07:41.938122 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f4a8c880313b406b4cd2b36e5344c75bdc8e7408a619bb8f538930f572d732\": container with ID starting with e1f4a8c880313b406b4cd2b36e5344c75bdc8e7408a619bb8f538930f572d732 not found: ID does not exist" containerID="e1f4a8c880313b406b4cd2b36e5344c75bdc8e7408a619bb8f538930f572d732" Oct 30 13:07:41.938186 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.938148 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f4a8c880313b406b4cd2b36e5344c75bdc8e7408a619bb8f538930f572d732"} err="failed to get container status \"e1f4a8c880313b406b4cd2b36e5344c75bdc8e7408a619bb8f538930f572d732\": rpc error: code = NotFound desc = could not find container \"e1f4a8c880313b406b4cd2b36e5344c75bdc8e7408a619bb8f538930f572d732\": container with ID starting with e1f4a8c880313b406b4cd2b36e5344c75bdc8e7408a619bb8f538930f572d732 not found: ID does not exist" Oct 30 13:07:41.978067 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.978023 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2"] Oct 30 13:07:41.980634 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:41.980601 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2675c0-openshift-default-766dbf4455-s42l2"] Oct 30 13:07:42.020963 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:42.020937 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:42.134708 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:42.134680 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:42.596721 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:42.596686 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:42.681220 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:42.681187 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" podUID="e2971b88-5b87-49db-b7f8-60c2a9d7116d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:42.764876 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:42.764842 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:42.884680 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:42.884643 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:43.031874 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:43.031834 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e43f0b-df10-4b4e-b7af-67d515178a84" path="/var/lib/kubelet/pods/75e43f0b-df10-4b4e-b7af-67d515178a84/volumes" Oct 30 13:07:43.080995 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:43.080959 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:43.216214 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:43.216134 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:43.743262 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:43.743215 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:44.659716 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:44.659677 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:45.171679 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.171653 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:45.326288 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.326054 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" Oct 30 13:07:45.347586 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.347563 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" Oct 30 13:07:45.354728 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.354705 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4"] Oct 30 13:07:45.363448 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.363399 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0818b575-9f29-4241-967b-448947aaf982-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0818b575-9f29-4241-967b-448947aaf982-plugins-config-volume\") pod \"0818b575-9f29-4241-967b-448947aaf982\" (UID: \"0818b575-9f29-4241-967b-448947aaf982\") " Oct 30 13:07:45.363700 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.363680 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0818b575-9f29-4241-967b-448947aaf982-kube-api-access-p875z\" (UniqueName: \"kubernetes.io/projected/0818b575-9f29-4241-967b-448947aaf982-kube-api-access-p875z\") pod \"0818b575-9f29-4241-967b-448947aaf982\" (UID: \"0818b575-9f29-4241-967b-448947aaf982\") " Oct 30 13:07:45.363831 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.363739 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0818b575-9f29-4241-967b-448947aaf982-plugins-config-volume" pod "0818b575-9f29-4241-967b-448947aaf982" (UID: "0818b575-9f29-4241-967b-448947aaf982"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:45.364733 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.364266 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0818b575-9f29-4241-967b-448947aaf982-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:45.366132 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.366109 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0818b575-9f29-4241-967b-448947aaf982-kube-api-access-p875z" pod "0818b575-9f29-4241-967b-448947aaf982" (UID: "0818b575-9f29-4241-967b-448947aaf982"). InnerVolumeSpecName "kube-api-access-p875z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:45.464903 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.464874 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2971b88-5b87-49db-b7f8-60c2a9d7116d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2971b88-5b87-49db-b7f8-60c2a9d7116d-plugins-config-volume\") pod \"e2971b88-5b87-49db-b7f8-60c2a9d7116d\" (UID: \"e2971b88-5b87-49db-b7f8-60c2a9d7116d\") " Oct 30 13:07:45.465080 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.465021 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2971b88-5b87-49db-b7f8-60c2a9d7116d-kube-api-access-twf4d\" (UniqueName: \"kubernetes.io/projected/e2971b88-5b87-49db-b7f8-60c2a9d7116d-kube-api-access-twf4d\") pod \"e2971b88-5b87-49db-b7f8-60c2a9d7116d\" (UID: \"e2971b88-5b87-49db-b7f8-60c2a9d7116d\") " Oct 30 13:07:45.465347 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.465322 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2971b88-5b87-49db-b7f8-60c2a9d7116d-plugins-config-volume" pod "e2971b88-5b87-49db-b7f8-60c2a9d7116d" (UID: "e2971b88-5b87-49db-b7f8-60c2a9d7116d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:45.465455 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.465406 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p875z\" (UniqueName: \"kubernetes.io/projected/0818b575-9f29-4241-967b-448947aaf982-kube-api-access-p875z\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:45.467385 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.467361 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2971b88-5b87-49db-b7f8-60c2a9d7116d-kube-api-access-twf4d" pod "e2971b88-5b87-49db-b7f8-60c2a9d7116d" (UID: "e2971b88-5b87-49db-b7f8-60c2a9d7116d"). InnerVolumeSpecName "kube-api-access-twf4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:45.472904 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.472883 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:45.566759 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.566666 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2971b88-5b87-49db-b7f8-60c2a9d7116d-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:45.566759 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.566701 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twf4d\" (UniqueName: \"kubernetes.io/projected/e2971b88-5b87-49db-b7f8-60c2a9d7116d-kube-api-access-twf4d\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:45.588094 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.588065 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:45.945027 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.944993 4413 generic.go:356] "Generic (PLEG): container finished" podID="0818b575-9f29-4241-967b-448947aaf982" containerID="a58fa4287c50b9b20a822260f3e9dd75c85630bc58d1298d813a70f9ba900471" exitCode=0 Oct 30 13:07:45.945223 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.945032 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" Oct 30 13:07:45.945223 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.945071 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" event={"ID":"0818b575-9f29-4241-967b-448947aaf982","Type":"ContainerDied","Data":"a58fa4287c50b9b20a822260f3e9dd75c85630bc58d1298d813a70f9ba900471"} Oct 30 13:07:45.945223 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.945121 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj" event={"ID":"0818b575-9f29-4241-967b-448947aaf982","Type":"ContainerDied","Data":"a906e8d35bf20c1ef635424a681772beb876cb0286775417c9d301e503122d70"} Oct 30 13:07:45.945223 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.945155 4413 scope.go:122] "RemoveContainer" containerID="a58fa4287c50b9b20a822260f3e9dd75c85630bc58d1298d813a70f9ba900471" Oct 30 13:07:45.946549 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.946525 4413 generic.go:356] "Generic (PLEG): container finished" podID="e2971b88-5b87-49db-b7f8-60c2a9d7116d" containerID="27f9a0f03583d6206ee5731025fcc540132eb075f406148a0b57c4860f39036d" exitCode=0 Oct 30 13:07:45.946671 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.946556 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" event={"ID":"e2971b88-5b87-49db-b7f8-60c2a9d7116d","Type":"ContainerDied","Data":"27f9a0f03583d6206ee5731025fcc540132eb075f406148a0b57c4860f39036d"} Oct 30 13:07:45.946671 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.946582 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" Oct 30 13:07:45.946671 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.946608 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl" event={"ID":"e2971b88-5b87-49db-b7f8-60c2a9d7116d","Type":"ContainerDied","Data":"54d54e6bd74c1c53f08c54aa134985156909cdee0de22c5ee128b87ebbbebe30"} Oct 30 13:07:45.955791 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.955769 4413 scope.go:122] "RemoveContainer" containerID="a58fa4287c50b9b20a822260f3e9dd75c85630bc58d1298d813a70f9ba900471" Oct 30 13:07:45.956072 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:07:45.956052 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a58fa4287c50b9b20a822260f3e9dd75c85630bc58d1298d813a70f9ba900471\": container with ID starting with a58fa4287c50b9b20a822260f3e9dd75c85630bc58d1298d813a70f9ba900471 not found: ID does not exist" containerID="a58fa4287c50b9b20a822260f3e9dd75c85630bc58d1298d813a70f9ba900471" Oct 30 13:07:45.956166 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.956079 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58fa4287c50b9b20a822260f3e9dd75c85630bc58d1298d813a70f9ba900471"} err="failed to get container status \"a58fa4287c50b9b20a822260f3e9dd75c85630bc58d1298d813a70f9ba900471\": rpc error: code = NotFound desc = could not find container \"a58fa4287c50b9b20a822260f3e9dd75c85630bc58d1298d813a70f9ba900471\": container with ID starting with a58fa4287c50b9b20a822260f3e9dd75c85630bc58d1298d813a70f9ba900471 not found: ID does not exist" Oct 30 13:07:45.956166 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.956100 4413 scope.go:122] "RemoveContainer" containerID="27f9a0f03583d6206ee5731025fcc540132eb075f406148a0b57c4860f39036d" Oct 30 13:07:45.994465 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.977484 4413 scope.go:122] "RemoveContainer" containerID="27f9a0f03583d6206ee5731025fcc540132eb075f406148a0b57c4860f39036d" Oct 30 13:07:45.994465 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:07:45.979904 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f9a0f03583d6206ee5731025fcc540132eb075f406148a0b57c4860f39036d\": container with ID starting with 27f9a0f03583d6206ee5731025fcc540132eb075f406148a0b57c4860f39036d not found: ID does not exist" containerID="27f9a0f03583d6206ee5731025fcc540132eb075f406148a0b57c4860f39036d" Oct 30 13:07:45.994465 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:45.979942 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f9a0f03583d6206ee5731025fcc540132eb075f406148a0b57c4860f39036d"} err="failed to get container status \"27f9a0f03583d6206ee5731025fcc540132eb075f406148a0b57c4860f39036d\": rpc error: code = NotFound desc = could not find container \"27f9a0f03583d6206ee5731025fcc540132eb075f406148a0b57c4860f39036d\": container with ID starting with 27f9a0f03583d6206ee5731025fcc540132eb075f406148a0b57c4860f39036d not found: ID does not exist" Oct 30 13:07:46.008453 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:46.008417 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj"] Oct 30 13:07:46.011579 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:46.011550 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-812990-74fdfd6586-9mkcj"] Oct 30 13:07:46.044212 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:46.044188 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl"] Oct 30 13:07:46.048142 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:46.048110 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-812990-cb87f9bb7-x9pdl"] Oct 30 13:07:46.160075 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:46.160051 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj"] Oct 30 13:07:46.198951 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:46.198866 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86"] Oct 30 13:07:46.318894 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:46.318863 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:46.427987 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:46.427957 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:46.945506 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:46.945478 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j"] Oct 30 13:07:47.019367 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:47.019338 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0818b575-9f29-4241-967b-448947aaf982" path="/var/lib/kubelet/pods/0818b575-9f29-4241-967b-448947aaf982/volumes" Oct 30 13:07:47.019735 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:47.019723 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2971b88-5b87-49db-b7f8-60c2a9d7116d" path="/var/lib/kubelet/pods/e2971b88-5b87-49db-b7f8-60c2a9d7116d/volumes" Oct 30 13:07:47.111870 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:47.111824 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:47.261633 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:47.261541 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:48.563988 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:48.563949 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:49.030287 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:49.030225 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:49.169276 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:49.169235 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:49.286872 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:49.286804 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:49.709652 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:49.709604 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:50.356888 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.356857 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:50.357059 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.357018 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" podUID="70146efe-2585-4812-8761-50cd5c027a1c" containerName="payload-pre-processing" containerID="cri-o://7d87ddf9a1b368f6cda798187caf2d874b97e15c8a8de581f31c0360844c55d6" gracePeriod=25 Oct 30 13:07:50.428511 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.428488 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" Oct 30 13:07:50.454422 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.454402 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" Oct 30 13:07:50.502787 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.501902 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:50.537637 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.537600 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68a11193-ed13-452f-b0b1-eaeb60b21465-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a11193-ed13-452f-b0b1-eaeb60b21465-plugins-config-volume\") pod \"68a11193-ed13-452f-b0b1-eaeb60b21465\" (UID: \"68a11193-ed13-452f-b0b1-eaeb60b21465\") " Oct 30 13:07:50.537788 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.537699 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdabf3ad-4ef3-4f8e-9f53-6565dacf505e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdabf3ad-4ef3-4f8e-9f53-6565dacf505e-plugins-config-volume\") pod \"fdabf3ad-4ef3-4f8e-9f53-6565dacf505e\" (UID: \"fdabf3ad-4ef3-4f8e-9f53-6565dacf505e\") " Oct 30 13:07:50.537788 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.537742 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdabf3ad-4ef3-4f8e-9f53-6565dacf505e-kube-api-access-6dmvb\" (UniqueName: \"kubernetes.io/projected/fdabf3ad-4ef3-4f8e-9f53-6565dacf505e-kube-api-access-6dmvb\") pod \"fdabf3ad-4ef3-4f8e-9f53-6565dacf505e\" (UID: \"fdabf3ad-4ef3-4f8e-9f53-6565dacf505e\") " Oct 30 13:07:50.537917 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.537820 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68a11193-ed13-452f-b0b1-eaeb60b21465-kube-api-access-vrlnm\" (UniqueName: \"kubernetes.io/projected/68a11193-ed13-452f-b0b1-eaeb60b21465-kube-api-access-vrlnm\") pod \"68a11193-ed13-452f-b0b1-eaeb60b21465\" (UID: \"68a11193-ed13-452f-b0b1-eaeb60b21465\") " Oct 30 13:07:50.538064 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.538039 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a11193-ed13-452f-b0b1-eaeb60b21465-plugins-config-volume" pod "68a11193-ed13-452f-b0b1-eaeb60b21465" (UID: "68a11193-ed13-452f-b0b1-eaeb60b21465"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:50.539799 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.538125 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdabf3ad-4ef3-4f8e-9f53-6565dacf505e-plugins-config-volume" pod "fdabf3ad-4ef3-4f8e-9f53-6565dacf505e" (UID: "fdabf3ad-4ef3-4f8e-9f53-6565dacf505e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:50.539799 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.538391 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a11193-ed13-452f-b0b1-eaeb60b21465-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:50.539799 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.538413 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdabf3ad-4ef3-4f8e-9f53-6565dacf505e-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:50.540293 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.540266 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdabf3ad-4ef3-4f8e-9f53-6565dacf505e-kube-api-access-6dmvb" pod "fdabf3ad-4ef3-4f8e-9f53-6565dacf505e" (UID: "fdabf3ad-4ef3-4f8e-9f53-6565dacf505e"). InnerVolumeSpecName "kube-api-access-6dmvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:50.540407 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.540380 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a11193-ed13-452f-b0b1-eaeb60b21465-kube-api-access-vrlnm" pod "68a11193-ed13-452f-b0b1-eaeb60b21465" (UID: "68a11193-ed13-452f-b0b1-eaeb60b21465"). InnerVolumeSpecName "kube-api-access-vrlnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:50.639863 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.639835 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dmvb\" (UniqueName: \"kubernetes.io/projected/fdabf3ad-4ef3-4f8e-9f53-6565dacf505e-kube-api-access-6dmvb\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:50.639863 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.639863 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrlnm\" (UniqueName: \"kubernetes.io/projected/68a11193-ed13-452f-b0b1-eaeb60b21465-kube-api-access-vrlnm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:50.693997 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.693928 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:50.766067 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.766036 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt"] Oct 30 13:07:50.766595 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.766302 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" podUID="951adfc0-673a-4716-b07a-e94182a7e608" containerName="maas-api" containerID="cri-o://08352913c8cf32e6ea451ba4708321fe79b1a9c6126fb8a76e946fdd3fcf80f0" gracePeriod=30 Oct 30 13:07:50.843541 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.843511 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:50.986639 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.986540 4413 generic.go:356] "Generic (PLEG): container finished" podID="951adfc0-673a-4716-b07a-e94182a7e608" containerID="08352913c8cf32e6ea451ba4708321fe79b1a9c6126fb8a76e946fdd3fcf80f0" exitCode=0 Oct 30 13:07:50.986639 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.986625 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" event={"ID":"951adfc0-673a-4716-b07a-e94182a7e608","Type":"ContainerDied","Data":"08352913c8cf32e6ea451ba4708321fe79b1a9c6126fb8a76e946fdd3fcf80f0"} Oct 30 13:07:50.991709 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.991681 4413 generic.go:356] "Generic (PLEG): container finished" podID="68a11193-ed13-452f-b0b1-eaeb60b21465" containerID="46bc5abd3a94b3010a0f89c1cf734cadbd3fe16580474ee674b503025f9e3226" exitCode=0 Oct 30 13:07:50.991890 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.991741 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" Oct 30 13:07:50.991890 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.991822 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" event={"ID":"68a11193-ed13-452f-b0b1-eaeb60b21465","Type":"ContainerDied","Data":"46bc5abd3a94b3010a0f89c1cf734cadbd3fe16580474ee674b503025f9e3226"} Oct 30 13:07:50.992351 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.991916 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b" event={"ID":"68a11193-ed13-452f-b0b1-eaeb60b21465","Type":"ContainerDied","Data":"b3d8a53970c706207082b1354b55cc50fb8cb9dd5671bfdbc884e3a50f53b863"} Oct 30 13:07:50.992351 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.991951 4413 scope.go:122] "RemoveContainer" containerID="46bc5abd3a94b3010a0f89c1cf734cadbd3fe16580474ee674b503025f9e3226" Oct 30 13:07:50.994062 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.994029 4413 generic.go:356] "Generic (PLEG): container finished" podID="fdabf3ad-4ef3-4f8e-9f53-6565dacf505e" containerID="baccd1ee02487a3f1a999ea59dfe371f52d2dfa486530e92d16cd2d3fd6f869d" exitCode=0 Oct 30 13:07:50.994172 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.994089 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" event={"ID":"fdabf3ad-4ef3-4f8e-9f53-6565dacf505e","Type":"ContainerDied","Data":"baccd1ee02487a3f1a999ea59dfe371f52d2dfa486530e92d16cd2d3fd6f869d"} Oct 30 13:07:50.994172 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.994129 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" event={"ID":"fdabf3ad-4ef3-4f8e-9f53-6565dacf505e","Type":"ContainerDied","Data":"c1df53b5c3bc110edf5d0f388d9ed7426f138e96c0130feee6d22c53d1310303"} Oct 30 13:07:50.994342 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:50.994200 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2" Oct 30 13:07:51.028075 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.026408 4413 scope.go:122] "RemoveContainer" containerID="46bc5abd3a94b3010a0f89c1cf734cadbd3fe16580474ee674b503025f9e3226" Oct 30 13:07:51.045372 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:07:51.036233 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46bc5abd3a94b3010a0f89c1cf734cadbd3fe16580474ee674b503025f9e3226\": container with ID starting with 46bc5abd3a94b3010a0f89c1cf734cadbd3fe16580474ee674b503025f9e3226 not found: ID does not exist" containerID="46bc5abd3a94b3010a0f89c1cf734cadbd3fe16580474ee674b503025f9e3226" Oct 30 13:07:51.045372 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.036294 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bc5abd3a94b3010a0f89c1cf734cadbd3fe16580474ee674b503025f9e3226"} err="failed to get container status \"46bc5abd3a94b3010a0f89c1cf734cadbd3fe16580474ee674b503025f9e3226\": rpc error: code = NotFound desc = could not find container \"46bc5abd3a94b3010a0f89c1cf734cadbd3fe16580474ee674b503025f9e3226\": container with ID starting with 46bc5abd3a94b3010a0f89c1cf734cadbd3fe16580474ee674b503025f9e3226 not found: ID does not exist" Oct 30 13:07:51.045372 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.036319 4413 scope.go:122] "RemoveContainer" containerID="baccd1ee02487a3f1a999ea59dfe371f52d2dfa486530e92d16cd2d3fd6f869d" Oct 30 13:07:51.055693 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.055668 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" Oct 30 13:07:51.056039 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.056020 4413 scope.go:122] "RemoveContainer" containerID="baccd1ee02487a3f1a999ea59dfe371f52d2dfa486530e92d16cd2d3fd6f869d" Oct 30 13:07:51.056354 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:07:51.056332 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baccd1ee02487a3f1a999ea59dfe371f52d2dfa486530e92d16cd2d3fd6f869d\": container with ID starting with baccd1ee02487a3f1a999ea59dfe371f52d2dfa486530e92d16cd2d3fd6f869d not found: ID does not exist" containerID="baccd1ee02487a3f1a999ea59dfe371f52d2dfa486530e92d16cd2d3fd6f869d" Oct 30 13:07:51.056435 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.056365 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baccd1ee02487a3f1a999ea59dfe371f52d2dfa486530e92d16cd2d3fd6f869d"} err="failed to get container status \"baccd1ee02487a3f1a999ea59dfe371f52d2dfa486530e92d16cd2d3fd6f869d\": rpc error: code = NotFound desc = could not find container \"baccd1ee02487a3f1a999ea59dfe371f52d2dfa486530e92d16cd2d3fd6f869d\": container with ID starting with baccd1ee02487a3f1a999ea59dfe371f52d2dfa486530e92d16cd2d3fd6f869d not found: ID does not exist" Oct 30 13:07:51.083417 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.083388 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2"] Oct 30 13:07:51.090508 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.090487 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2675c0-6bf8c6bc8c-gvpb2"] Oct 30 13:07:51.128935 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.128909 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b"] Oct 30 13:07:51.131442 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.131422 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2675c0-5c696f55dd-whl8b"] Oct 30 13:07:51.145795 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.145768 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/951adfc0-673a-4716-b07a-e94182a7e608-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951adfc0-673a-4716-b07a-e94182a7e608-openshift-service-ca\") pod \"951adfc0-673a-4716-b07a-e94182a7e608\" (UID: \"951adfc0-673a-4716-b07a-e94182a7e608\") " Oct 30 13:07:51.145983 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.145823 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/951adfc0-673a-4716-b07a-e94182a7e608-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951adfc0-673a-4716-b07a-e94182a7e608-maas-api-tls\") pod \"951adfc0-673a-4716-b07a-e94182a7e608\" (UID: \"951adfc0-673a-4716-b07a-e94182a7e608\") " Oct 30 13:07:51.145983 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.145866 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/951adfc0-673a-4716-b07a-e94182a7e608-metrics-certs\" (UniqueName: \"kubernetes.io/secret/951adfc0-673a-4716-b07a-e94182a7e608-metrics-certs\") pod \"951adfc0-673a-4716-b07a-e94182a7e608\" (UID: \"951adfc0-673a-4716-b07a-e94182a7e608\") " Oct 30 13:07:51.145983 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.145915 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951adfc0-673a-4716-b07a-e94182a7e608-kube-api-access-kh6gq\" (UniqueName: \"kubernetes.io/projected/951adfc0-673a-4716-b07a-e94182a7e608-kube-api-access-kh6gq\") pod \"951adfc0-673a-4716-b07a-e94182a7e608\" (UID: \"951adfc0-673a-4716-b07a-e94182a7e608\") " Oct 30 13:07:51.146749 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.146726 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951adfc0-673a-4716-b07a-e94182a7e608-openshift-service-ca" pod "951adfc0-673a-4716-b07a-e94182a7e608" (UID: "951adfc0-673a-4716-b07a-e94182a7e608"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:07:51.164991 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.164327 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" podUID="99b68128-c265-4be3-a666-d3e73ad7fa39" containerName="payload-processing" containerID="cri-o://b579fb1e4ec28095fa019f6c88664f5f513df6b4bfb53b5695247a1728b9bcf0" gracePeriod=25 Oct 30 13:07:51.164991 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.164429 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/951adfc0-673a-4716-b07a-e94182a7e608-metrics-certs" pod "951adfc0-673a-4716-b07a-e94182a7e608" (UID: "951adfc0-673a-4716-b07a-e94182a7e608"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:51.164991 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.164504 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/951adfc0-673a-4716-b07a-e94182a7e608-maas-api-tls" pod "951adfc0-673a-4716-b07a-e94182a7e608" (UID: "951adfc0-673a-4716-b07a-e94182a7e608"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:07:51.164991 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.164622 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951adfc0-673a-4716-b07a-e94182a7e608-kube-api-access-kh6gq" pod "951adfc0-673a-4716-b07a-e94182a7e608" (UID: "951adfc0-673a-4716-b07a-e94182a7e608"). InnerVolumeSpecName "kube-api-access-kh6gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:07:51.199865 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.199770 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" podUID="d3832da5-41b0-4993-b87d-971b41984eb3" containerName="payload-pre-processing" containerID="cri-o://15d39e98ed09e8be209cf62513f65bb624b8cc4a7928ab679c478a004e9d1a7b" gracePeriod=25 Oct 30 13:07:51.247279 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.247190 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951adfc0-673a-4716-b07a-e94182a7e608-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:51.247279 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.247216 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/951adfc0-673a-4716-b07a-e94182a7e608-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:51.247279 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.247227 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh6gq\" (UniqueName: \"kubernetes.io/projected/951adfc0-673a-4716-b07a-e94182a7e608-kube-api-access-kh6gq\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:51.247279 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.247237 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951adfc0-673a-4716-b07a-e94182a7e608-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:07:51.365875 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.365845 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:51.498884 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.498809 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:51.613239 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.613192 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:51.946730 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:51.946688 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" podUID="9aa641d6-2556-4e48-9503-85d33c43df8a" containerName="payload-processing" containerID="cri-o://505bbe69e570b309b94ca53bf22dd8b49763e6d65deae1c45bb13e0533ec59fb" gracePeriod=25 Oct 30 13:07:52.002622 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:52.002598 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" Oct 30 13:07:52.002781 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:52.002614 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt" event={"ID":"951adfc0-673a-4716-b07a-e94182a7e608","Type":"ContainerDied","Data":"7e6481676d8371ab84c42f65e9138e146d7bb47779f4c335df3d029605329346"} Oct 30 13:07:52.002781 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:52.002719 4413 scope.go:122] "RemoveContainer" containerID="08352913c8cf32e6ea451ba4708321fe79b1a9c6126fb8a76e946fdd3fcf80f0" Oct 30 13:07:52.085478 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:52.085458 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt"] Oct 30 13:07:52.097047 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:52.097025 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-42220c-84c6b48fb5-xn5vt"] Oct 30 13:07:52.118329 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:52.118301 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:53.019014 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:53.018980 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68a11193-ed13-452f-b0b1-eaeb60b21465" path="/var/lib/kubelet/pods/68a11193-ed13-452f-b0b1-eaeb60b21465/volumes" Oct 30 13:07:53.019544 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:53.019526 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="951adfc0-673a-4716-b07a-e94182a7e608" path="/var/lib/kubelet/pods/951adfc0-673a-4716-b07a-e94182a7e608/volumes" Oct 30 13:07:53.019888 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:53.019876 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdabf3ad-4ef3-4f8e-9f53-6565dacf505e" path="/var/lib/kubelet/pods/fdabf3ad-4ef3-4f8e-9f53-6565dacf505e/volumes" Oct 30 13:07:53.438118 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:53.438079 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:54.694383 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:54.694349 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" podUID="d3832da5-41b0-4993-b87d-971b41984eb3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:54.694773 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:54.694349 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" podUID="99b68128-c265-4be3-a666-d3e73ad7fa39" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:55.126841 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:55.126812 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:55.290225 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:55.290188 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" podUID="70146efe-2585-4812-8761-50cd5c027a1c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:56.663088 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:56.663055 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:56.790911 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:56.790878 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:56.921300 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:56.921218 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:57.099656 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:57.099621 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:58.664382 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:58.664330 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:07:59.084998 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:59.084916 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" podUID="9aa641d6-2556-4e48-9503-85d33c43df8a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:07:59.972317 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:07:59.972274 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:00.106130 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:00.106102 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:00.296565 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:00.296479 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q"] Oct 30 13:08:00.296814 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:00.296783 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" podUID="0e06b3dc-2413-4f9f-ba0d-691519d450f1" containerName="istio-proxy" containerID="cri-o://99d1b70dffabf794317ce94654a3e6839a16e16fbe0571dc1ae99d153bfa977d" gracePeriod=30 Oct 30 13:08:00.352295 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:00.352228 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:00.485870 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:00.485839 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:00.858348 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:00.858304 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:01.012876 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:01.012842 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:01.127715 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:01.127652 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:01.248543 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:01.248512 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:01.653874 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:01.653842 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:03.099135 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:03.099096 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:03.432585 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:03.432545 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:04.649499 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:04.649454 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:04.694562 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:04.694524 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" podUID="99b68128-c265-4be3-a666-d3e73ad7fa39" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:04.694776 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:04.694606 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" podUID="d3832da5-41b0-4993-b87d-971b41984eb3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:05.290403 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.290370 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" podUID="70146efe-2585-4812-8761-50cd5c027a1c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:05.554344 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.554290 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 13:08:05.708573 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.708542 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-envoy\") pod \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " Oct 30 13:08:05.708573 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.708580 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istiod-ca-cert\") pod \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " Oct 30 13:08:05.709111 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.708604 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-data\") pod \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " Oct 30 13:08:05.709111 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.708632 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-workload-socket\") pod \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " Oct 30 13:08:05.709111 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.708684 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e06b3dc-2413-4f9f-ba0d-691519d450f1-kube-api-access-qm97j\" (UniqueName: \"kubernetes.io/projected/0e06b3dc-2413-4f9f-ba0d-691519d450f1-kube-api-access-qm97j\") pod \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " Oct 30 13:08:05.709111 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.708761 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-token\" (UniqueName: \"kubernetes.io/projected/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-token\") pod \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " Oct 30 13:08:05.709111 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.708791 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-workload-certs\") pod \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " Oct 30 13:08:05.709111 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.708832 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-credential-socket\") pod \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " Oct 30 13:08:05.709111 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.708903 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-data" pod "0e06b3dc-2413-4f9f-ba0d-691519d450f1" (UID: "0e06b3dc-2413-4f9f-ba0d-691519d450f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:05.709111 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.708916 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-podinfo\") pod \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\" (UID: \"0e06b3dc-2413-4f9f-ba0d-691519d450f1\") " Oct 30 13:08:05.709111 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.709074 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-workload-socket" pod "0e06b3dc-2413-4f9f-ba0d-691519d450f1" (UID: "0e06b3dc-2413-4f9f-ba0d-691519d450f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:05.709111 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.709091 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-workload-certs" pod "0e06b3dc-2413-4f9f-ba0d-691519d450f1" (UID: "0e06b3dc-2413-4f9f-ba0d-691519d450f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:05.709770 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.709315 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-credential-socket" pod "0e06b3dc-2413-4f9f-ba0d-691519d450f1" (UID: "0e06b3dc-2413-4f9f-ba0d-691519d450f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:05.709770 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.709569 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:05.709770 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.709590 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:05.709770 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.709614 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:05.709770 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.709629 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:05.710038 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.709844 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istiod-ca-cert" pod "0e06b3dc-2413-4f9f-ba0d-691519d450f1" (UID: "0e06b3dc-2413-4f9f-ba0d-691519d450f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:05.711219 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.711193 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-envoy" pod "0e06b3dc-2413-4f9f-ba0d-691519d450f1" (UID: "0e06b3dc-2413-4f9f-ba0d-691519d450f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:05.711500 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.711388 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-token" pod "0e06b3dc-2413-4f9f-ba0d-691519d450f1" (UID: "0e06b3dc-2413-4f9f-ba0d-691519d450f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:05.711757 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.711734 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-podinfo" pod "0e06b3dc-2413-4f9f-ba0d-691519d450f1" (UID: "0e06b3dc-2413-4f9f-ba0d-691519d450f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:08:05.711883 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.711764 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e06b3dc-2413-4f9f-ba0d-691519d450f1-kube-api-access-qm97j" pod "0e06b3dc-2413-4f9f-ba0d-691519d450f1" (UID: "0e06b3dc-2413-4f9f-ba0d-691519d450f1"). InnerVolumeSpecName "kube-api-access-qm97j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:05.773727 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.773696 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp"] Oct 30 13:08:05.810965 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.810891 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:05.810965 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.810921 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:05.810965 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.810934 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:05.810965 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.810947 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm97j\" (UniqueName: \"kubernetes.io/projected/0e06b3dc-2413-4f9f-ba0d-691519d450f1-kube-api-access-qm97j\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:05.810965 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.810958 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e06b3dc-2413-4f9f-ba0d-691519d450f1-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:05.813315 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:08:05.813283 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:08:05.879695 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:05.879670 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:06.115243 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:06.115162 4413 generic.go:356] "Generic (PLEG): container finished" podID="0e06b3dc-2413-4f9f-ba0d-691519d450f1" containerID="99d1b70dffabf794317ce94654a3e6839a16e16fbe0571dc1ae99d153bfa977d" exitCode=0 Oct 30 13:08:06.115243 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:06.115231 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" Oct 30 13:08:06.115455 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:06.115281 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" event={"ID":"0e06b3dc-2413-4f9f-ba0d-691519d450f1","Type":"ContainerDied","Data":"99d1b70dffabf794317ce94654a3e6839a16e16fbe0571dc1ae99d153bfa977d"} Oct 30 13:08:06.115455 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:06.115342 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q" event={"ID":"0e06b3dc-2413-4f9f-ba0d-691519d450f1","Type":"ContainerDied","Data":"dd51b0c4d34a149d3817b88e27f4f3ffe57f0975c8f0760531c38745c5ecc798"} Oct 30 13:08:06.115455 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:06.115366 4413 scope.go:122] "RemoveContainer" containerID="99d1b70dffabf794317ce94654a3e6839a16e16fbe0571dc1ae99d153bfa977d" Oct 30 13:08:06.131354 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:06.131336 4413 scope.go:122] "RemoveContainer" containerID="95201af750515388c1419366eacbcbe11349a59bee05d1ad6ae6865bc422bf9a" Oct 30 13:08:06.140679 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:06.140664 4413 scope.go:122] "RemoveContainer" containerID="99d1b70dffabf794317ce94654a3e6839a16e16fbe0571dc1ae99d153bfa977d" Oct 30 13:08:06.140918 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:08:06.140901 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d1b70dffabf794317ce94654a3e6839a16e16fbe0571dc1ae99d153bfa977d\": container with ID starting with 99d1b70dffabf794317ce94654a3e6839a16e16fbe0571dc1ae99d153bfa977d not found: ID does not exist" containerID="99d1b70dffabf794317ce94654a3e6839a16e16fbe0571dc1ae99d153bfa977d" Oct 30 13:08:06.140962 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:06.140925 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d1b70dffabf794317ce94654a3e6839a16e16fbe0571dc1ae99d153bfa977d"} err="failed to get container status \"99d1b70dffabf794317ce94654a3e6839a16e16fbe0571dc1ae99d153bfa977d\": rpc error: code = NotFound desc = could not find container \"99d1b70dffabf794317ce94654a3e6839a16e16fbe0571dc1ae99d153bfa977d\": container with ID starting with 99d1b70dffabf794317ce94654a3e6839a16e16fbe0571dc1ae99d153bfa977d not found: ID does not exist" Oct 30 13:08:06.140962 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:06.140940 4413 scope.go:122] "RemoveContainer" containerID="95201af750515388c1419366eacbcbe11349a59bee05d1ad6ae6865bc422bf9a" Oct 30 13:08:06.141167 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:08:06.141151 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95201af750515388c1419366eacbcbe11349a59bee05d1ad6ae6865bc422bf9a\": container with ID starting with 95201af750515388c1419366eacbcbe11349a59bee05d1ad6ae6865bc422bf9a not found: ID does not exist" containerID="95201af750515388c1419366eacbcbe11349a59bee05d1ad6ae6865bc422bf9a" Oct 30 13:08:06.141209 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:06.141174 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95201af750515388c1419366eacbcbe11349a59bee05d1ad6ae6865bc422bf9a"} err="failed to get container status \"95201af750515388c1419366eacbcbe11349a59bee05d1ad6ae6865bc422bf9a\": rpc error: code = NotFound desc = could not find container \"95201af750515388c1419366eacbcbe11349a59bee05d1ad6ae6865bc422bf9a\": container with ID starting with 95201af750515388c1419366eacbcbe11349a59bee05d1ad6ae6865bc422bf9a not found: ID does not exist" Oct 30 13:08:06.172476 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:06.172455 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q"] Oct 30 13:08:06.174532 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:06.174513 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-42220c-openshift-default-69fcbd8647-kjt4q"] Oct 30 13:08:06.463920 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:06.463888 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:07.021680 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:07.021649 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e06b3dc-2413-4f9f-ba0d-691519d450f1" path="/var/lib/kubelet/pods/0e06b3dc-2413-4f9f-ba0d-691519d450f1/volumes" Oct 30 13:08:08.079563 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:08.079525 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq"] Oct 30 13:08:08.108707 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:08.108674 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh"] Oct 30 13:08:08.227570 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:08.227531 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:08.342199 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:08.342107 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:08.999201 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:08.999172 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:09.085303 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:09.085274 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" podUID="9aa641d6-2556-4e48-9503-85d33c43df8a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:09.140949 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:09.140926 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:09.335772 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:09.335680 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:09.782410 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:09.782373 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:09.986315 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:09.986288 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:10.623091 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:10.623038 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" Oct 30 13:08:10.661352 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:10.661327 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70146efe-2585-4812-8761-50cd5c027a1c-kube-api-access-grfvx\" (UniqueName: \"kubernetes.io/projected/70146efe-2585-4812-8761-50cd5c027a1c-kube-api-access-grfvx\") pod \"70146efe-2585-4812-8761-50cd5c027a1c\" (UID: \"70146efe-2585-4812-8761-50cd5c027a1c\") " Oct 30 13:08:10.661450 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:10.661367 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70146efe-2585-4812-8761-50cd5c027a1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70146efe-2585-4812-8761-50cd5c027a1c-plugins-config-volume\") pod \"70146efe-2585-4812-8761-50cd5c027a1c\" (UID: \"70146efe-2585-4812-8761-50cd5c027a1c\") " Oct 30 13:08:10.661743 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:10.661721 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70146efe-2585-4812-8761-50cd5c027a1c-plugins-config-volume" pod "70146efe-2585-4812-8761-50cd5c027a1c" (UID: "70146efe-2585-4812-8761-50cd5c027a1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:10.663482 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:10.663465 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70146efe-2585-4812-8761-50cd5c027a1c-kube-api-access-grfvx" pod "70146efe-2585-4812-8761-50cd5c027a1c" (UID: "70146efe-2585-4812-8761-50cd5c027a1c"). InnerVolumeSpecName "kube-api-access-grfvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:10.762407 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:10.762370 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grfvx\" (UniqueName: \"kubernetes.io/projected/70146efe-2585-4812-8761-50cd5c027a1c-kube-api-access-grfvx\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:10.762407 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:10.762399 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70146efe-2585-4812-8761-50cd5c027a1c-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:11.151382 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.151339 4413 generic.go:356] "Generic (PLEG): container finished" podID="70146efe-2585-4812-8761-50cd5c027a1c" containerID="7d87ddf9a1b368f6cda798187caf2d874b97e15c8a8de581f31c0360844c55d6" exitCode=0 Oct 30 13:08:11.151554 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.151386 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" Oct 30 13:08:11.151554 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.151407 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" event={"ID":"70146efe-2585-4812-8761-50cd5c027a1c","Type":"ContainerDied","Data":"7d87ddf9a1b368f6cda798187caf2d874b97e15c8a8de581f31c0360844c55d6"} Oct 30 13:08:11.151554 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.151461 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4" event={"ID":"70146efe-2585-4812-8761-50cd5c027a1c","Type":"ContainerDied","Data":"501f3415a2ff11a166ed0cb61cb8ccd898c4bd15a53f1d09c5d65899111a826e"} Oct 30 13:08:11.151554 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.151489 4413 scope.go:122] "RemoveContainer" containerID="7d87ddf9a1b368f6cda798187caf2d874b97e15c8a8de581f31c0360844c55d6" Oct 30 13:08:11.160842 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.160824 4413 scope.go:122] "RemoveContainer" containerID="7d87ddf9a1b368f6cda798187caf2d874b97e15c8a8de581f31c0360844c55d6" Oct 30 13:08:11.161105 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:08:11.161084 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d87ddf9a1b368f6cda798187caf2d874b97e15c8a8de581f31c0360844c55d6\": container with ID starting with 7d87ddf9a1b368f6cda798187caf2d874b97e15c8a8de581f31c0360844c55d6 not found: ID does not exist" containerID="7d87ddf9a1b368f6cda798187caf2d874b97e15c8a8de581f31c0360844c55d6" Oct 30 13:08:11.161182 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.161116 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d87ddf9a1b368f6cda798187caf2d874b97e15c8a8de581f31c0360844c55d6"} err="failed to get container status \"7d87ddf9a1b368f6cda798187caf2d874b97e15c8a8de581f31c0360844c55d6\": rpc error: code = NotFound desc = could not find container \"7d87ddf9a1b368f6cda798187caf2d874b97e15c8a8de581f31c0360844c55d6\": container with ID starting with 7d87ddf9a1b368f6cda798187caf2d874b97e15c8a8de581f31c0360844c55d6 not found: ID does not exist" Oct 30 13:08:11.203147 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.203122 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4"] Oct 30 13:08:11.214925 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.214903 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-42220c-8659d6cd87-6l5x4"] Oct 30 13:08:11.460782 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.460757 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" Oct 30 13:08:11.469769 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.469747 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99b68128-c265-4be3-a666-d3e73ad7fa39-kube-api-access-xf9sr\" (UniqueName: \"kubernetes.io/projected/99b68128-c265-4be3-a666-d3e73ad7fa39-kube-api-access-xf9sr\") pod \"99b68128-c265-4be3-a666-d3e73ad7fa39\" (UID: \"99b68128-c265-4be3-a666-d3e73ad7fa39\") " Oct 30 13:08:11.469872 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.469844 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99b68128-c265-4be3-a666-d3e73ad7fa39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99b68128-c265-4be3-a666-d3e73ad7fa39-plugins-config-volume\") pod \"99b68128-c265-4be3-a666-d3e73ad7fa39\" (UID: \"99b68128-c265-4be3-a666-d3e73ad7fa39\") " Oct 30 13:08:11.470189 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.470168 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b68128-c265-4be3-a666-d3e73ad7fa39-plugins-config-volume" pod "99b68128-c265-4be3-a666-d3e73ad7fa39" (UID: "99b68128-c265-4be3-a666-d3e73ad7fa39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:11.472010 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.471990 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b68128-c265-4be3-a666-d3e73ad7fa39-kube-api-access-xf9sr" pod "99b68128-c265-4be3-a666-d3e73ad7fa39" (UID: "99b68128-c265-4be3-a666-d3e73ad7fa39"). InnerVolumeSpecName "kube-api-access-xf9sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:11.496689 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.496671 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" Oct 30 13:08:11.571382 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.571297 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3832da5-41b0-4993-b87d-971b41984eb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3832da5-41b0-4993-b87d-971b41984eb3-plugins-config-volume\") pod \"d3832da5-41b0-4993-b87d-971b41984eb3\" (UID: \"d3832da5-41b0-4993-b87d-971b41984eb3\") " Oct 30 13:08:11.571555 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.571385 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3832da5-41b0-4993-b87d-971b41984eb3-kube-api-access-w2trv\" (UniqueName: \"kubernetes.io/projected/d3832da5-41b0-4993-b87d-971b41984eb3-kube-api-access-w2trv\") pod \"d3832da5-41b0-4993-b87d-971b41984eb3\" (UID: \"d3832da5-41b0-4993-b87d-971b41984eb3\") " Oct 30 13:08:11.571755 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.571733 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3832da5-41b0-4993-b87d-971b41984eb3-plugins-config-volume" pod "d3832da5-41b0-4993-b87d-971b41984eb3" (UID: "d3832da5-41b0-4993-b87d-971b41984eb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:11.571870 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.571856 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf9sr\" (UniqueName: \"kubernetes.io/projected/99b68128-c265-4be3-a666-d3e73ad7fa39-kube-api-access-xf9sr\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:11.571937 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.571872 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99b68128-c265-4be3-a666-d3e73ad7fa39-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:11.571937 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.571881 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3832da5-41b0-4993-b87d-971b41984eb3-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:11.574497 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.574468 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3832da5-41b0-4993-b87d-971b41984eb3-kube-api-access-w2trv" pod "d3832da5-41b0-4993-b87d-971b41984eb3" (UID: "d3832da5-41b0-4993-b87d-971b41984eb3"). InnerVolumeSpecName "kube-api-access-w2trv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:11.658301 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.658243 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:11.672775 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.672747 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2trv\" (UniqueName: \"kubernetes.io/projected/d3832da5-41b0-4993-b87d-971b41984eb3-kube-api-access-w2trv\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:11.976192 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:11.976158 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:12.102073 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.102041 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:12.166060 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.166020 4413 generic.go:356] "Generic (PLEG): container finished" podID="d3832da5-41b0-4993-b87d-971b41984eb3" containerID="15d39e98ed09e8be209cf62513f65bb624b8cc4a7928ab679c478a004e9d1a7b" exitCode=0 Oct 30 13:08:12.166278 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.166071 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" Oct 30 13:08:12.166278 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.166097 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" event={"ID":"d3832da5-41b0-4993-b87d-971b41984eb3","Type":"ContainerDied","Data":"15d39e98ed09e8be209cf62513f65bb624b8cc4a7928ab679c478a004e9d1a7b"} Oct 30 13:08:12.166278 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.166164 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86" event={"ID":"d3832da5-41b0-4993-b87d-971b41984eb3","Type":"ContainerDied","Data":"c16aa38714f14891e17688b3a564c2d703634b8944354f309854195a3662086e"} Oct 30 13:08:12.166278 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.166197 4413 scope.go:122] "RemoveContainer" containerID="15d39e98ed09e8be209cf62513f65bb624b8cc4a7928ab679c478a004e9d1a7b" Oct 30 13:08:12.168068 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.168042 4413 generic.go:356] "Generic (PLEG): container finished" podID="99b68128-c265-4be3-a666-d3e73ad7fa39" containerID="b579fb1e4ec28095fa019f6c88664f5f513df6b4bfb53b5695247a1728b9bcf0" exitCode=0 Oct 30 13:08:12.168184 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.168134 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" event={"ID":"99b68128-c265-4be3-a666-d3e73ad7fa39","Type":"ContainerDied","Data":"b579fb1e4ec28095fa019f6c88664f5f513df6b4bfb53b5695247a1728b9bcf0"} Oct 30 13:08:12.168184 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.168172 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" event={"ID":"99b68128-c265-4be3-a666-d3e73ad7fa39","Type":"ContainerDied","Data":"172a9de5ca88320a1d62cb94788aa836d8156931f180de374dcae0e89a7b9c49"} Oct 30 13:08:12.168325 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.168301 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj" Oct 30 13:08:12.178396 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.178377 4413 generic.go:356] "Generic (PLEG): container finished" podID="9aa641d6-2556-4e48-9503-85d33c43df8a" containerID="505bbe69e570b309b94ca53bf22dd8b49763e6d65deae1c45bb13e0533ec59fb" exitCode=0 Oct 30 13:08:12.178496 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.178461 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" event={"ID":"9aa641d6-2556-4e48-9503-85d33c43df8a","Type":"ContainerDied","Data":"505bbe69e570b309b94ca53bf22dd8b49763e6d65deae1c45bb13e0533ec59fb"} Oct 30 13:08:12.187960 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.187940 4413 scope.go:122] "RemoveContainer" containerID="15d39e98ed09e8be209cf62513f65bb624b8cc4a7928ab679c478a004e9d1a7b" Oct 30 13:08:12.188304 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:08:12.188273 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d39e98ed09e8be209cf62513f65bb624b8cc4a7928ab679c478a004e9d1a7b\": container with ID starting with 15d39e98ed09e8be209cf62513f65bb624b8cc4a7928ab679c478a004e9d1a7b not found: ID does not exist" containerID="15d39e98ed09e8be209cf62513f65bb624b8cc4a7928ab679c478a004e9d1a7b" Oct 30 13:08:12.188400 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.188315 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d39e98ed09e8be209cf62513f65bb624b8cc4a7928ab679c478a004e9d1a7b"} err="failed to get container status \"15d39e98ed09e8be209cf62513f65bb624b8cc4a7928ab679c478a004e9d1a7b\": rpc error: code = NotFound desc = could not find container \"15d39e98ed09e8be209cf62513f65bb624b8cc4a7928ab679c478a004e9d1a7b\": container with ID starting with 15d39e98ed09e8be209cf62513f65bb624b8cc4a7928ab679c478a004e9d1a7b not found: ID does not exist" Oct 30 13:08:12.188400 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.188339 4413 scope.go:122] "RemoveContainer" containerID="b579fb1e4ec28095fa019f6c88664f5f513df6b4bfb53b5695247a1728b9bcf0" Oct 30 13:08:12.199587 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.199571 4413 scope.go:122] "RemoveContainer" containerID="b579fb1e4ec28095fa019f6c88664f5f513df6b4bfb53b5695247a1728b9bcf0" Oct 30 13:08:12.199861 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:08:12.199841 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b579fb1e4ec28095fa019f6c88664f5f513df6b4bfb53b5695247a1728b9bcf0\": container with ID starting with b579fb1e4ec28095fa019f6c88664f5f513df6b4bfb53b5695247a1728b9bcf0 not found: ID does not exist" containerID="b579fb1e4ec28095fa019f6c88664f5f513df6b4bfb53b5695247a1728b9bcf0" Oct 30 13:08:12.199960 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.199868 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b579fb1e4ec28095fa019f6c88664f5f513df6b4bfb53b5695247a1728b9bcf0"} err="failed to get container status \"b579fb1e4ec28095fa019f6c88664f5f513df6b4bfb53b5695247a1728b9bcf0\": rpc error: code = NotFound desc = could not find container \"b579fb1e4ec28095fa019f6c88664f5f513df6b4bfb53b5695247a1728b9bcf0\": container with ID starting with b579fb1e4ec28095fa019f6c88664f5f513df6b4bfb53b5695247a1728b9bcf0 not found: ID does not exist" Oct 30 13:08:12.202344 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.202324 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" Oct 30 13:08:12.229065 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.229014 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:12.237473 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.237447 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86"] Oct 30 13:08:12.247180 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.247157 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fddaa3-68994dbf87-j6x86"] Oct 30 13:08:12.277841 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.277815 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aa641d6-2556-4e48-9503-85d33c43df8a-kube-api-access-ms5tc\" (UniqueName: \"kubernetes.io/projected/9aa641d6-2556-4e48-9503-85d33c43df8a-kube-api-access-ms5tc\") pod \"9aa641d6-2556-4e48-9503-85d33c43df8a\" (UID: \"9aa641d6-2556-4e48-9503-85d33c43df8a\") " Oct 30 13:08:12.277998 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.277874 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aa641d6-2556-4e48-9503-85d33c43df8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa641d6-2556-4e48-9503-85d33c43df8a-plugins-config-volume\") pod \"9aa641d6-2556-4e48-9503-85d33c43df8a\" (UID: \"9aa641d6-2556-4e48-9503-85d33c43df8a\") " Oct 30 13:08:12.278301 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.278279 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa641d6-2556-4e48-9503-85d33c43df8a-plugins-config-volume" pod "9aa641d6-2556-4e48-9503-85d33c43df8a" (UID: "9aa641d6-2556-4e48-9503-85d33c43df8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:12.280113 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.280093 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa641d6-2556-4e48-9503-85d33c43df8a-kube-api-access-ms5tc" pod "9aa641d6-2556-4e48-9503-85d33c43df8a" (UID: "9aa641d6-2556-4e48-9503-85d33c43df8a"). InnerVolumeSpecName "kube-api-access-ms5tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:12.294500 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.294467 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj"] Oct 30 13:08:12.300407 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.300381 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fddaa3-68dc868d8-h9mbj"] Oct 30 13:08:12.363647 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.363619 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:12.379408 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.379386 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms5tc\" (UniqueName: \"kubernetes.io/projected/9aa641d6-2556-4e48-9503-85d33c43df8a-kube-api-access-ms5tc\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:12.379408 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.379409 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa641d6-2556-4e48-9503-85d33c43df8a-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:12.488810 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.488722 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:12.771489 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:12.771409 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:13.020082 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:13.020052 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70146efe-2585-4812-8761-50cd5c027a1c" path="/var/lib/kubelet/pods/70146efe-2585-4812-8761-50cd5c027a1c/volumes" Oct 30 13:08:13.020599 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:13.020582 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99b68128-c265-4be3-a666-d3e73ad7fa39" path="/var/lib/kubelet/pods/99b68128-c265-4be3-a666-d3e73ad7fa39/volumes" Oct 30 13:08:13.020995 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:13.020982 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3832da5-41b0-4993-b87d-971b41984eb3" path="/var/lib/kubelet/pods/d3832da5-41b0-4993-b87d-971b41984eb3/volumes" Oct 30 13:08:13.079843 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:13.079791 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" podUID="25322f93-360d-472f-8c1b-382c04505663" containerName="payload-processing" containerID="cri-o://88d673a10a442aa4ac5964d6c211158cb2ca45e80b66c3105693d43ec2f9c06b" gracePeriod=25 Oct 30 13:08:13.087380 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:08:13.087357 4413 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa641d6_2556_4e48_9503_85d33c43df8a.slice\": RecentStats: unable to find data in memory cache]" Oct 30 13:08:13.109378 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:13.109339 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" podUID="ad21c9c0-28e9-4e5c-b82b-64747d88bc93" containerName="payload-pre-processing" containerID="cri-o://f32d0b51f39127ed0d7682e98b7bd96ec9cfbfcb19c7de4f94c99906ba2e7250" gracePeriod=25 Oct 30 13:08:13.187193 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:13.187161 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" event={"ID":"9aa641d6-2556-4e48-9503-85d33c43df8a","Type":"ContainerDied","Data":"bb5e08473a8492bea608f4335a93d51fc1978b3ac34f5d2a5ee944ac2d3c92c1"} Oct 30 13:08:13.187361 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:13.187170 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j" Oct 30 13:08:13.187361 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:13.187278 4413 scope.go:122] "RemoveContainer" containerID="505bbe69e570b309b94ca53bf22dd8b49763e6d65deae1c45bb13e0533ec59fb" Oct 30 13:08:13.256902 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:13.256876 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j"] Oct 30 13:08:13.268383 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:13.268358 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9d45fb-5b5f557cb-lpd5j"] Oct 30 13:08:14.158147 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:14.158116 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:14.746935 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:14.746900 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2"] Oct 30 13:08:14.775108 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:14.775071 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn"] Oct 30 13:08:14.884314 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:14.884273 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:15.020661 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:15.020583 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aa641d6-2556-4e48-9503-85d33c43df8a" path="/var/lib/kubelet/pods/9aa641d6-2556-4e48-9503-85d33c43df8a/volumes" Oct 30 13:08:16.054682 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:16.054654 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:17.822348 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:17.822315 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:19.706608 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:19.706524 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:19.885522 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:19.885491 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" podUID="ad21c9c0-28e9-4e5c-b82b-64747d88bc93" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:20.524850 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:20.524817 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:20.620774 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:20.620742 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4"] Oct 30 13:08:20.675425 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:20.675392 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:20.794942 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:20.794846 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" containerID="cri-o://e82e4708918f6f4a2aaefb9cd588c0eca1a206ee7e54369034a464ade933d301" gracePeriod=45 Oct 30 13:08:20.863506 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:20.863475 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:20.890680 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:20.890651 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" podUID="25322f93-360d-472f-8c1b-382c04505663" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:21.087567 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.087538 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 13:08:21.180984 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.180942 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-model-cache\") pod \"36c57b1c-6aec-48f0-81aa-adac4196039e\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " Oct 30 13:08:21.181178 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.181063 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-home\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-home\") pod \"36c57b1c-6aec-48f0-81aa-adac4196039e\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " Oct 30 13:08:21.181178 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.181116 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-dshm\") pod \"36c57b1c-6aec-48f0-81aa-adac4196039e\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " Oct 30 13:08:21.181178 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.181152 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36c57b1c-6aec-48f0-81aa-adac4196039e-kube-api-access-slchw\" (UniqueName: \"kubernetes.io/projected/36c57b1c-6aec-48f0-81aa-adac4196039e-kube-api-access-slchw\") pod \"36c57b1c-6aec-48f0-81aa-adac4196039e\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " Oct 30 13:08:21.181391 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.181294 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36c57b1c-6aec-48f0-81aa-adac4196039e-tls-certs\" (UniqueName: \"kubernetes.io/secret/36c57b1c-6aec-48f0-81aa-adac4196039e-tls-certs\") pod \"36c57b1c-6aec-48f0-81aa-adac4196039e\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " Oct 30 13:08:21.181391 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.181341 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-tmp-dir\") pod \"36c57b1c-6aec-48f0-81aa-adac4196039e\" (UID: \"36c57b1c-6aec-48f0-81aa-adac4196039e\") " Oct 30 13:08:21.181970 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.181930 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-home" pod "36c57b1c-6aec-48f0-81aa-adac4196039e" (UID: "36c57b1c-6aec-48f0-81aa-adac4196039e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:21.182114 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.181969 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-tmp-dir" pod "36c57b1c-6aec-48f0-81aa-adac4196039e" (UID: "36c57b1c-6aec-48f0-81aa-adac4196039e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:21.182274 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.182221 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-model-cache" pod "36c57b1c-6aec-48f0-81aa-adac4196039e" (UID: "36c57b1c-6aec-48f0-81aa-adac4196039e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:21.183711 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.183680 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:21.184372 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.184348 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-dshm" pod "36c57b1c-6aec-48f0-81aa-adac4196039e" (UID: "36c57b1c-6aec-48f0-81aa-adac4196039e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:21.184459 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.184426 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c57b1c-6aec-48f0-81aa-adac4196039e-tls-certs" pod "36c57b1c-6aec-48f0-81aa-adac4196039e" (UID: "36c57b1c-6aec-48f0-81aa-adac4196039e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:21.185367 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.185347 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c57b1c-6aec-48f0-81aa-adac4196039e-kube-api-access-slchw" pod "36c57b1c-6aec-48f0-81aa-adac4196039e" (UID: "36c57b1c-6aec-48f0-81aa-adac4196039e"). InnerVolumeSpecName "kube-api-access-slchw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:21.233092 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.233062 4413 generic.go:356] "Generic (PLEG): container finished" podID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerID="e82e4708918f6f4a2aaefb9cd588c0eca1a206ee7e54369034a464ade933d301" exitCode=0 Oct 30 13:08:21.233225 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.233124 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" event={"ID":"36c57b1c-6aec-48f0-81aa-adac4196039e","Type":"ContainerDied","Data":"e82e4708918f6f4a2aaefb9cd588c0eca1a206ee7e54369034a464ade933d301"} Oct 30 13:08:21.233225 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.233162 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" event={"ID":"36c57b1c-6aec-48f0-81aa-adac4196039e","Type":"ContainerDied","Data":"6b95687352a450b4a86bde6477bedd53354bd7915728775190b3f42d2d153840"} Oct 30 13:08:21.233225 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.233178 4413 scope.go:122] "RemoveContainer" containerID="e82e4708918f6f4a2aaefb9cd588c0eca1a206ee7e54369034a464ade933d301" Oct 30 13:08:21.233225 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.233131 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp" Oct 30 13:08:21.247718 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.247695 4413 scope.go:122] "RemoveContainer" containerID="e82e4708918f6f4a2aaefb9cd588c0eca1a206ee7e54369034a464ade933d301" Oct 30 13:08:21.248010 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:08:21.247991 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e82e4708918f6f4a2aaefb9cd588c0eca1a206ee7e54369034a464ade933d301\": container with ID starting with e82e4708918f6f4a2aaefb9cd588c0eca1a206ee7e54369034a464ade933d301 not found: ID does not exist" containerID="e82e4708918f6f4a2aaefb9cd588c0eca1a206ee7e54369034a464ade933d301" Oct 30 13:08:21.248060 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.248020 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e82e4708918f6f4a2aaefb9cd588c0eca1a206ee7e54369034a464ade933d301"} err="failed to get container status \"e82e4708918f6f4a2aaefb9cd588c0eca1a206ee7e54369034a464ade933d301\": rpc error: code = NotFound desc = could not find container \"e82e4708918f6f4a2aaefb9cd588c0eca1a206ee7e54369034a464ade933d301\": container with ID starting with e82e4708918f6f4a2aaefb9cd588c0eca1a206ee7e54369034a464ade933d301 not found: ID does not exist" Oct 30 13:08:21.282993 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.282965 4413 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.282993 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.282995 4413 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.283186 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.283003 4413 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.283186 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.283012 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slchw\" (UniqueName: \"kubernetes.io/projected/36c57b1c-6aec-48f0-81aa-adac4196039e-kube-api-access-slchw\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.283186 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.283021 4413 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36c57b1c-6aec-48f0-81aa-adac4196039e-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.283186 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.283032 4413 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36c57b1c-6aec-48f0-81aa-adac4196039e-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:21.295994 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.295965 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp"] Oct 30 13:08:21.298707 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.298650 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-47fd99/simulator-w2-47fd99-kserve-55db6bdc4-dtzgp"] Oct 30 13:08:21.603608 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:21.603536 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:23.018820 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:23.018786 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" path="/var/lib/kubelet/pods/36c57b1c-6aec-48f0-81aa-adac4196039e/volumes" Oct 30 13:08:23.641742 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:23.641309 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:25.193915 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.193881 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt"] Oct 30 13:08:25.194368 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.194143 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" podUID="0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40" containerName="maas-api" containerID="cri-o://2fc26a6e03594db198f7b17bceee9099e2a388ecdc1cfad61f99b2c45e272a67" gracePeriod=30 Oct 30 13:08:25.241539 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.241515 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:25.455941 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.455883 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" Oct 30 13:08:25.525760 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.525730 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-metrics-certs\") pod \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\" (UID: \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\") " Oct 30 13:08:25.525943 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.525814 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-maas-api-tls\") pod \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\" (UID: \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\") " Oct 30 13:08:25.525943 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.525855 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-kube-api-access-t847b\" (UniqueName: \"kubernetes.io/projected/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-kube-api-access-t847b\") pod \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\" (UID: \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\") " Oct 30 13:08:25.525943 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.525884 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-openshift-service-ca\") pod \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\" (UID: \"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40\") " Oct 30 13:08:25.526355 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.526326 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-openshift-service-ca" pod "0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40" (UID: "0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:25.528075 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.528050 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-maas-api-tls" pod "0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40" (UID: "0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:25.528565 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.528548 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-metrics-certs" pod "0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40" (UID: "0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:25.528668 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.528639 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-kube-api-access-t847b" pod "0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40" (UID: "0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40"). InnerVolumeSpecName "kube-api-access-t847b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:25.621448 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.621381 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" podUID="40dac033-149e-423e-9630-baeb522263d0" containerName="payload-pre-processing" containerID="cri-o://0973a451e4cf823e4e5c558ce28ce829cc16ba036cc0d218ce264943cccf024f" gracePeriod=25 Oct 30 13:08:25.628380 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.627855 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:25.628380 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.627889 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:25.628380 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.627904 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t847b\" (UniqueName: \"kubernetes.io/projected/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-kube-api-access-t847b\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:25.628380 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.627935 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:25.831868 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.831790 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:25.979879 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:25.979846 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:26.120202 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:26.120126 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:26.263531 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:26.263502 4413 generic.go:356] "Generic (PLEG): container finished" podID="0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40" containerID="2fc26a6e03594db198f7b17bceee9099e2a388ecdc1cfad61f99b2c45e272a67" exitCode=0 Oct 30 13:08:26.264022 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:26.263550 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" Oct 30 13:08:26.264022 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:26.263585 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" event={"ID":"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40","Type":"ContainerDied","Data":"2fc26a6e03594db198f7b17bceee9099e2a388ecdc1cfad61f99b2c45e272a67"} Oct 30 13:08:26.264022 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:26.263644 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt" event={"ID":"0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40","Type":"ContainerDied","Data":"48c61fdb0e5ab02cb0315422804e32bf51438787c20b0ca5b990272119af7ad5"} Oct 30 13:08:26.264022 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:26.263668 4413 scope.go:122] "RemoveContainer" containerID="2fc26a6e03594db198f7b17bceee9099e2a388ecdc1cfad61f99b2c45e272a67" Oct 30 13:08:26.273651 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:26.273633 4413 scope.go:122] "RemoveContainer" containerID="2fc26a6e03594db198f7b17bceee9099e2a388ecdc1cfad61f99b2c45e272a67" Oct 30 13:08:26.273935 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:08:26.273916 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc26a6e03594db198f7b17bceee9099e2a388ecdc1cfad61f99b2c45e272a67\": container with ID starting with 2fc26a6e03594db198f7b17bceee9099e2a388ecdc1cfad61f99b2c45e272a67 not found: ID does not exist" containerID="2fc26a6e03594db198f7b17bceee9099e2a388ecdc1cfad61f99b2c45e272a67" Oct 30 13:08:26.273986 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:26.273945 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc26a6e03594db198f7b17bceee9099e2a388ecdc1cfad61f99b2c45e272a67"} err="failed to get container status \"2fc26a6e03594db198f7b17bceee9099e2a388ecdc1cfad61f99b2c45e272a67\": rpc error: code = NotFound desc = could not find container \"2fc26a6e03594db198f7b17bceee9099e2a388ecdc1cfad61f99b2c45e272a67\": container with ID starting with 2fc26a6e03594db198f7b17bceee9099e2a388ecdc1cfad61f99b2c45e272a67 not found: ID does not exist" Oct 30 13:08:26.317787 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:26.317767 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt"] Oct 30 13:08:26.329549 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:26.329528 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-47fd99-86bb4cf7bc-xqnzt"] Oct 30 13:08:26.510960 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:26.510933 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:27.019918 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:27.019889 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40" path="/var/lib/kubelet/pods/0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40/volumes" Oct 30 13:08:27.785393 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:27.785360 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:27.952822 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:27.952787 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:29.659723 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:29.659685 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" podUID="40dac033-149e-423e-9630-baeb522263d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:29.768691 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:29.768647 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" containerID="cri-o://ae72f5d2c4876472bc7957ca6a58c934cc3436fe4e99a8bb3d74ac326d8e0f41" gracePeriod=45 Oct 30 13:08:29.769909 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:29.769881 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:29.795919 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:29.795876 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" podUID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" containerName="main" containerID="cri-o://4b0294837a8e0a91b10e05e94f9f81ef86a95a506fb7764fbfcd583e5798f892" gracePeriod=45 Oct 30 13:08:29.884703 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:29.884674 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" podUID="ad21c9c0-28e9-4e5c-b82b-64747d88bc93" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:30.050483 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.050453 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 13:08:30.087796 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.087772 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 13:08:30.177212 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.177146 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-dshm\") pod \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " Oct 30 13:08:30.177212 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.177209 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-tmp-dir\") pod \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " Oct 30 13:08:30.177571 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.177238 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-model-cache\") pod \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " Oct 30 13:08:30.177571 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.177279 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-tls-certs\") pod \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " Oct 30 13:08:30.177571 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.177318 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-dshm\") pod \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " Oct 30 13:08:30.177571 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.177403 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-home\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-home\") pod \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " Oct 30 13:08:30.177571 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.177466 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-model-cache\") pod \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " Oct 30 13:08:30.177571 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.177532 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-tls-certs\") pod \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " Oct 30 13:08:30.177571 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.177569 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-tmp-dir\") pod \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " Oct 30 13:08:30.178210 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.177623 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-kube-api-access-xsdtp\" (UniqueName: \"kubernetes.io/projected/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-kube-api-access-xsdtp\") pod \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " Oct 30 13:08:30.178210 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.177656 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-kube-api-access-cvq6t\" (UniqueName: \"kubernetes.io/projected/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-kube-api-access-cvq6t\") pod \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\" (UID: \"c8ea2a69-24a6-4f0b-adb0-06a662f793fa\") " Oct 30 13:08:30.178210 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.177725 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-home\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-home\") pod \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\" (UID: \"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90\") " Oct 30 13:08:30.178210 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.177475 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-tmp-dir" pod "c8ea2a69-24a6-4f0b-adb0-06a662f793fa" (UID: "c8ea2a69-24a6-4f0b-adb0-06a662f793fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:30.178210 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.177619 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-model-cache" pod "3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" (UID: "3d24c3bd-af3d-4379-a8bf-7f40a42b6e90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:30.178210 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.177719 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-home" pod "c8ea2a69-24a6-4f0b-adb0-06a662f793fa" (UID: "c8ea2a69-24a6-4f0b-adb0-06a662f793fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:30.178210 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.177727 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-model-cache" pod "c8ea2a69-24a6-4f0b-adb0-06a662f793fa" (UID: "c8ea2a69-24a6-4f0b-adb0-06a662f793fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:30.178625 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.178402 4413 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.178625 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.178424 4413 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.178625 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.178437 4413 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.178625 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.178450 4413 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.178844 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.178663 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-home" pod "3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" (UID: "3d24c3bd-af3d-4379-a8bf-7f40a42b6e90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:30.178844 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.178697 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-tmp-dir" pod "3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" (UID: "3d24c3bd-af3d-4379-a8bf-7f40a42b6e90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:30.180859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.180827 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-dshm" pod "3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" (UID: "3d24c3bd-af3d-4379-a8bf-7f40a42b6e90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:30.181241 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.180928 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-tls-certs" pod "c8ea2a69-24a6-4f0b-adb0-06a662f793fa" (UID: "c8ea2a69-24a6-4f0b-adb0-06a662f793fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:30.181241 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.180949 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-kube-api-access-xsdtp" pod "3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" (UID: "3d24c3bd-af3d-4379-a8bf-7f40a42b6e90"). InnerVolumeSpecName "kube-api-access-xsdtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:30.181241 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.181219 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-tls-certs" pod "3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" (UID: "3d24c3bd-af3d-4379-a8bf-7f40a42b6e90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:30.181526 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.181505 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-kube-api-access-cvq6t" pod "c8ea2a69-24a6-4f0b-adb0-06a662f793fa" (UID: "c8ea2a69-24a6-4f0b-adb0-06a662f793fa"). InnerVolumeSpecName "kube-api-access-cvq6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:30.182840 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.182817 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-dshm" pod "c8ea2a69-24a6-4f0b-adb0-06a662f793fa" (UID: "c8ea2a69-24a6-4f0b-adb0-06a662f793fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:30.279430 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.279397 4413 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.279430 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.279424 4413 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.279430 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.279433 4413 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.279651 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.279443 4413 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.279651 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.279458 4413 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.279651 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.279466 4413 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.279651 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.279475 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsdtp\" (UniqueName: \"kubernetes.io/projected/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90-kube-api-access-xsdtp\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.279651 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.279484 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvq6t\" (UniqueName: \"kubernetes.io/projected/c8ea2a69-24a6-4f0b-adb0-06a662f793fa-kube-api-access-cvq6t\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:30.286931 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.286896 4413 generic.go:356] "Generic (PLEG): container finished" podID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerID="ae72f5d2c4876472bc7957ca6a58c934cc3436fe4e99a8bb3d74ac326d8e0f41" exitCode=0 Oct 30 13:08:30.287055 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.286946 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" Oct 30 13:08:30.287055 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.286977 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" event={"ID":"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90","Type":"ContainerDied","Data":"ae72f5d2c4876472bc7957ca6a58c934cc3436fe4e99a8bb3d74ac326d8e0f41"} Oct 30 13:08:30.287055 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.287018 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2" event={"ID":"3d24c3bd-af3d-4379-a8bf-7f40a42b6e90","Type":"ContainerDied","Data":"00f7694d69e92e041bcf68d1439c5e1f137d65d4f39ba7a12e21d0e01991d56c"} Oct 30 13:08:30.287055 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.287039 4413 scope.go:122] "RemoveContainer" containerID="ae72f5d2c4876472bc7957ca6a58c934cc3436fe4e99a8bb3d74ac326d8e0f41" Oct 30 13:08:30.288405 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.288379 4413 generic.go:356] "Generic (PLEG): container finished" podID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" containerID="4b0294837a8e0a91b10e05e94f9f81ef86a95a506fb7764fbfcd583e5798f892" exitCode=0 Oct 30 13:08:30.288621 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.288406 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" event={"ID":"c8ea2a69-24a6-4f0b-adb0-06a662f793fa","Type":"ContainerDied","Data":"4b0294837a8e0a91b10e05e94f9f81ef86a95a506fb7764fbfcd583e5798f892"} Oct 30 13:08:30.288621 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.288450 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" event={"ID":"c8ea2a69-24a6-4f0b-adb0-06a662f793fa","Type":"ContainerDied","Data":"d9970e926a9fb3b28fec707e06ab3b581b2175cf5d4083a5e20e489734712212"} Oct 30 13:08:30.288621 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.288419 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn" Oct 30 13:08:30.297424 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.297407 4413 scope.go:122] "RemoveContainer" containerID="ae72f5d2c4876472bc7957ca6a58c934cc3436fe4e99a8bb3d74ac326d8e0f41" Oct 30 13:08:30.297765 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:08:30.297745 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae72f5d2c4876472bc7957ca6a58c934cc3436fe4e99a8bb3d74ac326d8e0f41\": container with ID starting with ae72f5d2c4876472bc7957ca6a58c934cc3436fe4e99a8bb3d74ac326d8e0f41 not found: ID does not exist" containerID="ae72f5d2c4876472bc7957ca6a58c934cc3436fe4e99a8bb3d74ac326d8e0f41" Oct 30 13:08:30.297854 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.297773 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae72f5d2c4876472bc7957ca6a58c934cc3436fe4e99a8bb3d74ac326d8e0f41"} err="failed to get container status \"ae72f5d2c4876472bc7957ca6a58c934cc3436fe4e99a8bb3d74ac326d8e0f41\": rpc error: code = NotFound desc = could not find container \"ae72f5d2c4876472bc7957ca6a58c934cc3436fe4e99a8bb3d74ac326d8e0f41\": container with ID starting with ae72f5d2c4876472bc7957ca6a58c934cc3436fe4e99a8bb3d74ac326d8e0f41 not found: ID does not exist" Oct 30 13:08:30.297854 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.297790 4413 scope.go:122] "RemoveContainer" containerID="4b0294837a8e0a91b10e05e94f9f81ef86a95a506fb7764fbfcd583e5798f892" Oct 30 13:08:30.306373 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.306356 4413 scope.go:122] "RemoveContainer" containerID="4b0294837a8e0a91b10e05e94f9f81ef86a95a506fb7764fbfcd583e5798f892" Oct 30 13:08:30.306636 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:08:30.306617 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b0294837a8e0a91b10e05e94f9f81ef86a95a506fb7764fbfcd583e5798f892\": container with ID starting with 4b0294837a8e0a91b10e05e94f9f81ef86a95a506fb7764fbfcd583e5798f892 not found: ID does not exist" containerID="4b0294837a8e0a91b10e05e94f9f81ef86a95a506fb7764fbfcd583e5798f892" Oct 30 13:08:30.306751 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.306646 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b0294837a8e0a91b10e05e94f9f81ef86a95a506fb7764fbfcd583e5798f892"} err="failed to get container status \"4b0294837a8e0a91b10e05e94f9f81ef86a95a506fb7764fbfcd583e5798f892\": rpc error: code = NotFound desc = could not find container \"4b0294837a8e0a91b10e05e94f9f81ef86a95a506fb7764fbfcd583e5798f892\": container with ID starting with 4b0294837a8e0a91b10e05e94f9f81ef86a95a506fb7764fbfcd583e5798f892 not found: ID does not exist" Oct 30 13:08:30.338894 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.338872 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn"] Oct 30 13:08:30.349557 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.349521 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/simulator-w3-6a7148-kserve-7c6887c6b9-bj6zn"] Oct 30 13:08:30.383535 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.383503 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2"] Oct 30 13:08:30.386902 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.386883 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-6a7148/distinct-w3-6a7148-kserve-57b64c974c-5bch2"] Oct 30 13:08:30.891370 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:30.891339 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" podUID="25322f93-360d-472f-8c1b-382c04505663" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:31.018564 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:31.018536 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" path="/var/lib/kubelet/pods/3d24c3bd-af3d-4379-a8bf-7f40a42b6e90/volumes" Oct 30 13:08:31.019033 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:31.019005 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" path="/var/lib/kubelet/pods/c8ea2a69-24a6-4f0b-adb0-06a662f793fa/volumes" Oct 30 13:08:31.452596 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:31.452565 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:32.223843 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:32.223813 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:32.350106 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:32.350077 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:32.994970 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:32.994942 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:33.309843 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.309809 4413 generic.go:356] "Generic (PLEG): container finished" podID="ad21c9c0-28e9-4e5c-b82b-64747d88bc93" containerID="f32d0b51f39127ed0d7682e98b7bd96ec9cfbfcb19c7de4f94c99906ba2e7250" exitCode=0 Oct 30 13:08:33.310313 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.309883 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" event={"ID":"ad21c9c0-28e9-4e5c-b82b-64747d88bc93","Type":"ContainerDied","Data":"f32d0b51f39127ed0d7682e98b7bd96ec9cfbfcb19c7de4f94c99906ba2e7250"} Oct 30 13:08:33.311954 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.311931 4413 generic.go:356] "Generic (PLEG): container finished" podID="25322f93-360d-472f-8c1b-382c04505663" containerID="88d673a10a442aa4ac5964d6c211158cb2ca45e80b66c3105693d43ec2f9c06b" exitCode=0 Oct 30 13:08:33.312087 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.312017 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" event={"ID":"25322f93-360d-472f-8c1b-382c04505663","Type":"ContainerDied","Data":"88d673a10a442aa4ac5964d6c211158cb2ca45e80b66c3105693d43ec2f9c06b"} Oct 30 13:08:33.391306 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.391277 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" Oct 30 13:08:33.408806 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.408788 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" Oct 30 13:08:33.533792 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.533721 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad21c9c0-28e9-4e5c-b82b-64747d88bc93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad21c9c0-28e9-4e5c-b82b-64747d88bc93-plugins-config-volume\") pod \"ad21c9c0-28e9-4e5c-b82b-64747d88bc93\" (UID: \"ad21c9c0-28e9-4e5c-b82b-64747d88bc93\") " Oct 30 13:08:33.533975 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.533884 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25322f93-360d-472f-8c1b-382c04505663-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25322f93-360d-472f-8c1b-382c04505663-plugins-config-volume\") pod \"25322f93-360d-472f-8c1b-382c04505663\" (UID: \"25322f93-360d-472f-8c1b-382c04505663\") " Oct 30 13:08:33.533975 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.533910 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25322f93-360d-472f-8c1b-382c04505663-kube-api-access-l9qxc\" (UniqueName: \"kubernetes.io/projected/25322f93-360d-472f-8c1b-382c04505663-kube-api-access-l9qxc\") pod \"25322f93-360d-472f-8c1b-382c04505663\" (UID: \"25322f93-360d-472f-8c1b-382c04505663\") " Oct 30 13:08:33.533975 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.533941 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad21c9c0-28e9-4e5c-b82b-64747d88bc93-kube-api-access-pp4d9\" (UniqueName: \"kubernetes.io/projected/ad21c9c0-28e9-4e5c-b82b-64747d88bc93-kube-api-access-pp4d9\") pod \"ad21c9c0-28e9-4e5c-b82b-64747d88bc93\" (UID: \"ad21c9c0-28e9-4e5c-b82b-64747d88bc93\") " Oct 30 13:08:33.534169 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.534138 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad21c9c0-28e9-4e5c-b82b-64747d88bc93-plugins-config-volume" pod "ad21c9c0-28e9-4e5c-b82b-64747d88bc93" (UID: "ad21c9c0-28e9-4e5c-b82b-64747d88bc93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:33.534297 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.534273 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25322f93-360d-472f-8c1b-382c04505663-plugins-config-volume" pod "25322f93-360d-472f-8c1b-382c04505663" (UID: "25322f93-360d-472f-8c1b-382c04505663"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:33.534382 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.534344 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad21c9c0-28e9-4e5c-b82b-64747d88bc93-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:33.536141 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.536119 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25322f93-360d-472f-8c1b-382c04505663-kube-api-access-l9qxc" pod "25322f93-360d-472f-8c1b-382c04505663" (UID: "25322f93-360d-472f-8c1b-382c04505663"). InnerVolumeSpecName "kube-api-access-l9qxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:33.536279 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.536243 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad21c9c0-28e9-4e5c-b82b-64747d88bc93-kube-api-access-pp4d9" pod "ad21c9c0-28e9-4e5c-b82b-64747d88bc93" (UID: "ad21c9c0-28e9-4e5c-b82b-64747d88bc93"). InnerVolumeSpecName "kube-api-access-pp4d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:33.598091 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.598050 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:33.635589 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.635555 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25322f93-360d-472f-8c1b-382c04505663-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:33.635589 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.635583 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9qxc\" (UniqueName: \"kubernetes.io/projected/25322f93-360d-472f-8c1b-382c04505663-kube-api-access-l9qxc\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:33.635589 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.635593 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp4d9\" (UniqueName: \"kubernetes.io/projected/ad21c9c0-28e9-4e5c-b82b-64747d88bc93-kube-api-access-pp4d9\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:33.821007 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:33.820911 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:34.318960 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:34.318936 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" Oct 30 13:08:34.319362 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:34.318977 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq" event={"ID":"25322f93-360d-472f-8c1b-382c04505663","Type":"ContainerDied","Data":"5e1c65e3176f0d2d088fd7a8c845e7ef623325ef7021c76e93e7219d8f623bf0"} Oct 30 13:08:34.319362 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:34.319038 4413 scope.go:122] "RemoveContainer" containerID="88d673a10a442aa4ac5964d6c211158cb2ca45e80b66c3105693d43ec2f9c06b" Oct 30 13:08:34.320629 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:34.320601 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" event={"ID":"ad21c9c0-28e9-4e5c-b82b-64747d88bc93","Type":"ContainerDied","Data":"2db54fb79f6ec91f07a016251c2209e127b9c34cd165167d2a670823a76fa1c6"} Oct 30 13:08:34.320748 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:34.320631 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh" Oct 30 13:08:34.329198 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:34.329173 4413 scope.go:122] "RemoveContainer" containerID="f32d0b51f39127ed0d7682e98b7bd96ec9cfbfcb19c7de4f94c99906ba2e7250" Oct 30 13:08:34.401127 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:34.401098 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh"] Oct 30 13:08:34.412786 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:34.412760 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-327e31-77c96b8867-9swbh"] Oct 30 13:08:34.452426 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:34.452397 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq"] Oct 30 13:08:34.458005 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:34.457983 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-327e31-7b6b59ff5d-flcqq"] Oct 30 13:08:34.495796 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:34.495771 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:35.018629 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:35.018597 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25322f93-360d-472f-8c1b-382c04505663" path="/var/lib/kubelet/pods/25322f93-360d-472f-8c1b-382c04505663/volumes" Oct 30 13:08:35.019102 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:35.019087 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad21c9c0-28e9-4e5c-b82b-64747d88bc93" path="/var/lib/kubelet/pods/ad21c9c0-28e9-4e5c-b82b-64747d88bc93/volumes" Oct 30 13:08:35.791694 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:35.791656 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:35.813959 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:08:35.813925 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:08:36.056543 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:36.056414 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:36.175362 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:36.175322 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:36.365180 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:36.365101 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps"] Oct 30 13:08:36.365574 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:36.365329 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" podUID="a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" containerName="istio-proxy" containerID="cri-o://e8e895b77f0f84ed9df1eaacf7692c7578f861388ccfec3b0c2124fd3ab02e05" gracePeriod=30 Oct 30 13:08:36.445029 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:36.444996 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:37.501870 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:37.501829 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:38.824366 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:38.824335 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp"] Oct 30 13:08:38.824811 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:38.824596 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" podUID="74ccd1d2-3901-46f4-b765-2ace6683fad1" containerName="maas-api" containerID="cri-o://6fa19483225f69b667b0a783c62b9968d9c6d384268c9e964aa235189fdb2018" gracePeriod=30 Oct 30 13:08:38.899029 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:38.898986 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:39.099293 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.099271 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 13:08:39.186684 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.186654 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ccd1d2-3901-46f4-b765-2ace6683fad1-kube-api-access-lm46c\" (UniqueName: \"kubernetes.io/projected/74ccd1d2-3901-46f4-b765-2ace6683fad1-kube-api-access-lm46c\") pod \"74ccd1d2-3901-46f4-b765-2ace6683fad1\" (UID: \"74ccd1d2-3901-46f4-b765-2ace6683fad1\") " Oct 30 13:08:39.186874 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.186769 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74ccd1d2-3901-46f4-b765-2ace6683fad1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74ccd1d2-3901-46f4-b765-2ace6683fad1-maas-api-tls\") pod \"74ccd1d2-3901-46f4-b765-2ace6683fad1\" (UID: \"74ccd1d2-3901-46f4-b765-2ace6683fad1\") " Oct 30 13:08:39.186874 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.186799 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74ccd1d2-3901-46f4-b765-2ace6683fad1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ccd1d2-3901-46f4-b765-2ace6683fad1-metrics-certs\") pod \"74ccd1d2-3901-46f4-b765-2ace6683fad1\" (UID: \"74ccd1d2-3901-46f4-b765-2ace6683fad1\") " Oct 30 13:08:39.187001 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.186916 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ccd1d2-3901-46f4-b765-2ace6683fad1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ccd1d2-3901-46f4-b765-2ace6683fad1-openshift-service-ca\") pod \"74ccd1d2-3901-46f4-b765-2ace6683fad1\" (UID: \"74ccd1d2-3901-46f4-b765-2ace6683fad1\") " Oct 30 13:08:39.187489 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.187298 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ccd1d2-3901-46f4-b765-2ace6683fad1-openshift-service-ca" pod "74ccd1d2-3901-46f4-b765-2ace6683fad1" (UID: "74ccd1d2-3901-46f4-b765-2ace6683fad1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:39.187831 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.187448 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ccd1d2-3901-46f4-b765-2ace6683fad1-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:39.189431 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.189406 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ccd1d2-3901-46f4-b765-2ace6683fad1-kube-api-access-lm46c" pod "74ccd1d2-3901-46f4-b765-2ace6683fad1" (UID: "74ccd1d2-3901-46f4-b765-2ace6683fad1"). InnerVolumeSpecName "kube-api-access-lm46c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:39.189529 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.189429 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74ccd1d2-3901-46f4-b765-2ace6683fad1-maas-api-tls" pod "74ccd1d2-3901-46f4-b765-2ace6683fad1" (UID: "74ccd1d2-3901-46f4-b765-2ace6683fad1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:39.189529 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.189450 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74ccd1d2-3901-46f4-b765-2ace6683fad1-metrics-certs" pod "74ccd1d2-3901-46f4-b765-2ace6683fad1" (UID: "74ccd1d2-3901-46f4-b765-2ace6683fad1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:08:39.288859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.288825 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm46c\" (UniqueName: \"kubernetes.io/projected/74ccd1d2-3901-46f4-b765-2ace6683fad1-kube-api-access-lm46c\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:39.288859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.288858 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74ccd1d2-3901-46f4-b765-2ace6683fad1-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:39.289062 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.288871 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ccd1d2-3901-46f4-b765-2ace6683fad1-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:39.353756 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.353675 4413 generic.go:356] "Generic (PLEG): container finished" podID="74ccd1d2-3901-46f4-b765-2ace6683fad1" containerID="6fa19483225f69b667b0a783c62b9968d9c6d384268c9e964aa235189fdb2018" exitCode=0 Oct 30 13:08:39.353756 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.353721 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" Oct 30 13:08:39.353756 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.353736 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" event={"ID":"74ccd1d2-3901-46f4-b765-2ace6683fad1","Type":"ContainerDied","Data":"6fa19483225f69b667b0a783c62b9968d9c6d384268c9e964aa235189fdb2018"} Oct 30 13:08:39.353999 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.353781 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp" event={"ID":"74ccd1d2-3901-46f4-b765-2ace6683fad1","Type":"ContainerDied","Data":"5145f13405115d087aa0cb053a17af2f8a0ea3a23224e00019b48a49e0d6829d"} Oct 30 13:08:39.353999 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.353803 4413 scope.go:122] "RemoveContainer" containerID="6fa19483225f69b667b0a783c62b9968d9c6d384268c9e964aa235189fdb2018" Oct 30 13:08:39.362989 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.362965 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:39.364066 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.364052 4413 scope.go:122] "RemoveContainer" containerID="6fa19483225f69b667b0a783c62b9968d9c6d384268c9e964aa235189fdb2018" Oct 30 13:08:39.364342 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:08:39.364324 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa19483225f69b667b0a783c62b9968d9c6d384268c9e964aa235189fdb2018\": container with ID starting with 6fa19483225f69b667b0a783c62b9968d9c6d384268c9e964aa235189fdb2018 not found: ID does not exist" containerID="6fa19483225f69b667b0a783c62b9968d9c6d384268c9e964aa235189fdb2018" Oct 30 13:08:39.364407 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.364349 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa19483225f69b667b0a783c62b9968d9c6d384268c9e964aa235189fdb2018"} err="failed to get container status \"6fa19483225f69b667b0a783c62b9968d9c6d384268c9e964aa235189fdb2018\": rpc error: code = NotFound desc = could not find container \"6fa19483225f69b667b0a783c62b9968d9c6d384268c9e964aa235189fdb2018\": container with ID starting with 6fa19483225f69b667b0a783c62b9968d9c6d384268c9e964aa235189fdb2018 not found: ID does not exist" Oct 30 13:08:39.406708 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.406679 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp"] Oct 30 13:08:39.412781 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.412754 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-6a7148-6dcc6697d6-c9njp"] Oct 30 13:08:39.476794 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.476761 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:39.660020 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:39.659975 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" podUID="40dac033-149e-423e-9630-baeb522263d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 13:08:40.147147 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:40.147119 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:41.019237 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.019207 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ccd1d2-3901-46f4-b765-2ace6683fad1" path="/var/lib/kubelet/pods/74ccd1d2-3901-46f4-b765-2ace6683fad1/volumes" Oct 30 13:08:41.040339 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.040304 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:41.098681 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.098652 4413 patch_prober.go:29] interesting pod/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 13:08:41.098815 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.098695 4413 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" podUID="a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 13:08:41.235864 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.235826 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:41.353744 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.353662 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh"] Oct 30 13:08:41.353995 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.353856 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" podUID="adb76aa8-2ad3-4bb6-bea9-ae31de43322e" containerName="istio-proxy" containerID="cri-o://21c846ad397bb2cc4fc7816ee0ece9a035a4200cedd7662c2acbb9dcbd21929f" gracePeriod=30 Oct 30 13:08:41.374887 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.374859 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:41.488033 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.488008 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:41.622720 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.622695 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 13:08:41.712344 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.712304 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istiod-ca-cert\") pod \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " Oct 30 13:08:41.712559 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.712374 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-credential-socket\") pod \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " Oct 30 13:08:41.712559 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.712398 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-token\" (UniqueName: \"kubernetes.io/projected/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-token\") pod \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " Oct 30 13:08:41.712559 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.712422 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-envoy\") pod \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " Oct 30 13:08:41.712559 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.712470 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-workload-socket\") pod \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " Oct 30 13:08:41.712559 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.712561 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-podinfo\") pod \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " Oct 30 13:08:41.712887 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.712598 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-workload-certs\") pod \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " Oct 30 13:08:41.712887 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.712635 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-kube-api-access-dghxf\" (UniqueName: \"kubernetes.io/projected/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-kube-api-access-dghxf\") pod \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " Oct 30 13:08:41.712887 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.712676 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-data\") pod \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\" (UID: \"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7\") " Oct 30 13:08:41.712887 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.712733 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-credential-socket" pod "a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" (UID: "a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:41.713132 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.712897 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-workload-socket" pod "a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" (UID: "a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:41.713132 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.712913 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-workload-certs" pod "a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" (UID: "a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:41.713132 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.713018 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-data" pod "a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" (UID: "a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:41.713132 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.713118 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istiod-ca-cert" pod "a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" (UID: "a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:41.713457 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.713412 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:41.713654 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.713631 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:41.713735 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.713661 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:41.713735 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.713678 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:41.713735 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.713692 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:41.715013 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.714992 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-token" pod "a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" (UID: "a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:41.715124 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.715003 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-podinfo" pod "a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" (UID: "a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:08:41.715452 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.715428 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-kube-api-access-dghxf" pod "a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" (UID: "a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7"). InnerVolumeSpecName "kube-api-access-dghxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:41.715452 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.715441 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-envoy" pod "a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" (UID: "a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:41.815067 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.815030 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dghxf\" (UniqueName: \"kubernetes.io/projected/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-kube-api-access-dghxf\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:41.815067 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.815061 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:41.815067 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.815072 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:41.815434 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.815080 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:41.856435 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:41.856375 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:42.374091 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:42.374052 4413 generic.go:356] "Generic (PLEG): container finished" podID="a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" containerID="e8e895b77f0f84ed9df1eaacf7692c7578f861388ccfec3b0c2124fd3ab02e05" exitCode=0 Oct 30 13:08:42.374615 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:42.374102 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" Oct 30 13:08:42.374615 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:42.374138 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" event={"ID":"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7","Type":"ContainerDied","Data":"e8e895b77f0f84ed9df1eaacf7692c7578f861388ccfec3b0c2124fd3ab02e05"} Oct 30 13:08:42.374615 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:42.374191 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps" event={"ID":"a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7","Type":"ContainerDied","Data":"92b36127afd714b0b052168d18028ad125ca46e13aa194924e96e4b96d7e4c51"} Oct 30 13:08:42.374615 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:42.374214 4413 scope.go:122] "RemoveContainer" containerID="e8e895b77f0f84ed9df1eaacf7692c7578f861388ccfec3b0c2124fd3ab02e05" Oct 30 13:08:42.384502 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:42.384485 4413 scope.go:122] "RemoveContainer" containerID="765b3b9754ffe9d27b216e47c9aaaa2a4b858d3f07e4d43374c4916f36eb29ab" Oct 30 13:08:42.393197 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:42.393179 4413 scope.go:122] "RemoveContainer" containerID="e8e895b77f0f84ed9df1eaacf7692c7578f861388ccfec3b0c2124fd3ab02e05" Oct 30 13:08:42.393492 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:08:42.393472 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e895b77f0f84ed9df1eaacf7692c7578f861388ccfec3b0c2124fd3ab02e05\": container with ID starting with e8e895b77f0f84ed9df1eaacf7692c7578f861388ccfec3b0c2124fd3ab02e05 not found: ID does not exist" containerID="e8e895b77f0f84ed9df1eaacf7692c7578f861388ccfec3b0c2124fd3ab02e05" Oct 30 13:08:42.393544 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:42.393500 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e895b77f0f84ed9df1eaacf7692c7578f861388ccfec3b0c2124fd3ab02e05"} err="failed to get container status \"e8e895b77f0f84ed9df1eaacf7692c7578f861388ccfec3b0c2124fd3ab02e05\": rpc error: code = NotFound desc = could not find container \"e8e895b77f0f84ed9df1eaacf7692c7578f861388ccfec3b0c2124fd3ab02e05\": container with ID starting with e8e895b77f0f84ed9df1eaacf7692c7578f861388ccfec3b0c2124fd3ab02e05 not found: ID does not exist" Oct 30 13:08:42.393544 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:42.393517 4413 scope.go:122] "RemoveContainer" containerID="765b3b9754ffe9d27b216e47c9aaaa2a4b858d3f07e4d43374c4916f36eb29ab" Oct 30 13:08:42.393765 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:08:42.393749 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765b3b9754ffe9d27b216e47c9aaaa2a4b858d3f07e4d43374c4916f36eb29ab\": container with ID starting with 765b3b9754ffe9d27b216e47c9aaaa2a4b858d3f07e4d43374c4916f36eb29ab not found: ID does not exist" containerID="765b3b9754ffe9d27b216e47c9aaaa2a4b858d3f07e4d43374c4916f36eb29ab" Oct 30 13:08:42.393808 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:42.393771 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765b3b9754ffe9d27b216e47c9aaaa2a4b858d3f07e4d43374c4916f36eb29ab"} err="failed to get container status \"765b3b9754ffe9d27b216e47c9aaaa2a4b858d3f07e4d43374c4916f36eb29ab\": rpc error: code = NotFound desc = could not find container \"765b3b9754ffe9d27b216e47c9aaaa2a4b858d3f07e4d43374c4916f36eb29ab\": container with ID starting with 765b3b9754ffe9d27b216e47c9aaaa2a4b858d3f07e4d43374c4916f36eb29ab not found: ID does not exist" Oct 30 13:08:42.438516 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:42.438487 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps"] Oct 30 13:08:42.450284 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:42.450180 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-47fd99-openshift-default-75b64cdc59-7l9ps"] Oct 30 13:08:42.478602 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:42.478573 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:42.615452 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:42.615419 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:43.018104 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:43.018067 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" path="/var/lib/kubelet/pods/a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7/volumes" Oct 30 13:08:43.266876 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:43.266843 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:44.436313 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:44.436279 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:45.227218 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:45.227182 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:45.885173 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:45.885115 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" Oct 30 13:08:45.958333 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:45.958306 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40dac033-149e-423e-9630-baeb522263d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40dac033-149e-423e-9630-baeb522263d0-plugins-config-volume\") pod \"40dac033-149e-423e-9630-baeb522263d0\" (UID: \"40dac033-149e-423e-9630-baeb522263d0\") " Oct 30 13:08:45.958522 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:45.958431 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40dac033-149e-423e-9630-baeb522263d0-kube-api-access-mvd88\" (UniqueName: \"kubernetes.io/projected/40dac033-149e-423e-9630-baeb522263d0-kube-api-access-mvd88\") pod \"40dac033-149e-423e-9630-baeb522263d0\" (UID: \"40dac033-149e-423e-9630-baeb522263d0\") " Oct 30 13:08:45.958717 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:45.958693 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40dac033-149e-423e-9630-baeb522263d0-plugins-config-volume" pod "40dac033-149e-423e-9630-baeb522263d0" (UID: "40dac033-149e-423e-9630-baeb522263d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:45.960726 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:45.960696 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40dac033-149e-423e-9630-baeb522263d0-kube-api-access-mvd88" pod "40dac033-149e-423e-9630-baeb522263d0" (UID: "40dac033-149e-423e-9630-baeb522263d0"). InnerVolumeSpecName "kube-api-access-mvd88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:46.059678 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.059652 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvd88\" (UniqueName: \"kubernetes.io/projected/40dac033-149e-423e-9630-baeb522263d0-kube-api-access-mvd88\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:46.059678 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.059675 4413 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40dac033-149e-423e-9630-baeb522263d0-plugins-config-volume\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:46.397050 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.397023 4413 generic.go:356] "Generic (PLEG): container finished" podID="40dac033-149e-423e-9630-baeb522263d0" containerID="0973a451e4cf823e4e5c558ce28ce829cc16ba036cc0d218ce264943cccf024f" exitCode=0 Oct 30 13:08:46.397173 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.397067 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" Oct 30 13:08:46.397173 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.397100 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" event={"ID":"40dac033-149e-423e-9630-baeb522263d0","Type":"ContainerDied","Data":"0973a451e4cf823e4e5c558ce28ce829cc16ba036cc0d218ce264943cccf024f"} Oct 30 13:08:46.397173 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.397154 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4" event={"ID":"40dac033-149e-423e-9630-baeb522263d0","Type":"ContainerDied","Data":"23537721379d6431dc779a8fbbafb328225eb7dced50a80482534d659fb99c62"} Oct 30 13:08:46.397330 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.397179 4413 scope.go:122] "RemoveContainer" containerID="0973a451e4cf823e4e5c558ce28ce829cc16ba036cc0d218ce264943cccf024f" Oct 30 13:08:46.465407 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.465393 4413 scope.go:122] "RemoveContainer" containerID="0973a451e4cf823e4e5c558ce28ce829cc16ba036cc0d218ce264943cccf024f" Oct 30 13:08:46.465750 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:08:46.465726 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0973a451e4cf823e4e5c558ce28ce829cc16ba036cc0d218ce264943cccf024f\": container with ID starting with 0973a451e4cf823e4e5c558ce28ce829cc16ba036cc0d218ce264943cccf024f not found: ID does not exist" containerID="0973a451e4cf823e4e5c558ce28ce829cc16ba036cc0d218ce264943cccf024f" Oct 30 13:08:46.465825 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.465762 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0973a451e4cf823e4e5c558ce28ce829cc16ba036cc0d218ce264943cccf024f"} err="failed to get container status \"0973a451e4cf823e4e5c558ce28ce829cc16ba036cc0d218ce264943cccf024f\": rpc error: code = NotFound desc = could not find container \"0973a451e4cf823e4e5c558ce28ce829cc16ba036cc0d218ce264943cccf024f\": container with ID starting with 0973a451e4cf823e4e5c558ce28ce829cc16ba036cc0d218ce264943cccf024f not found: ID does not exist" Oct 30 13:08:46.471783 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.471760 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4"] Oct 30 13:08:46.483370 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.483347 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6a7148-6f4795df8c-kl6l4"] Oct 30 13:08:46.600434 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.600400 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 13:08:46.666723 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.666680 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-token\" (UniqueName: \"kubernetes.io/projected/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-token\") pod \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " Oct 30 13:08:46.666940 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.666747 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-podinfo\") pod \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " Oct 30 13:08:46.666940 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.666795 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-data\") pod \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " Oct 30 13:08:46.666940 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.666856 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-kube-api-access-m7684\" (UniqueName: \"kubernetes.io/projected/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-kube-api-access-m7684\") pod \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " Oct 30 13:08:46.666940 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.666885 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-envoy\") pod \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " Oct 30 13:08:46.666940 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.666932 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istiod-ca-cert\") pod \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " Oct 30 13:08:46.667242 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.666960 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-workload-certs\") pod \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " Oct 30 13:08:46.667242 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.667001 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-credential-socket\") pod \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " Oct 30 13:08:46.667242 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.667152 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-workload-socket\") pod \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\" (UID: \"adb76aa8-2ad3-4bb6-bea9-ae31de43322e\") " Oct 30 13:08:46.667242 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.667183 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-data" pod "adb76aa8-2ad3-4bb6-bea9-ae31de43322e" (UID: "adb76aa8-2ad3-4bb6-bea9-ae31de43322e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:46.667242 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.667433 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-workload-certs" pod "adb76aa8-2ad3-4bb6-bea9-ae31de43322e" (UID: "adb76aa8-2ad3-4bb6-bea9-ae31de43322e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:46.667242 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.667530 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-credential-socket" pod "adb76aa8-2ad3-4bb6-bea9-ae31de43322e" (UID: "adb76aa8-2ad3-4bb6-bea9-ae31de43322e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:46.667242 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.667742 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istiod-ca-cert" pod "adb76aa8-2ad3-4bb6-bea9-ae31de43322e" (UID: "adb76aa8-2ad3-4bb6-bea9-ae31de43322e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:08:46.667242 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.667794 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-workload-socket" pod "adb76aa8-2ad3-4bb6-bea9-ae31de43322e" (UID: "adb76aa8-2ad3-4bb6-bea9-ae31de43322e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:46.668428 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.667849 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:46.668428 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.667873 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:46.668428 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.667887 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:46.669373 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.669326 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-kube-api-access-m7684" pod "adb76aa8-2ad3-4bb6-bea9-ae31de43322e" (UID: "adb76aa8-2ad3-4bb6-bea9-ae31de43322e"). InnerVolumeSpecName "kube-api-access-m7684". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:46.669483 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.669428 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-podinfo" pod "adb76aa8-2ad3-4bb6-bea9-ae31de43322e" (UID: "adb76aa8-2ad3-4bb6-bea9-ae31de43322e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:08:46.669709 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.669693 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-token" pod "adb76aa8-2ad3-4bb6-bea9-ae31de43322e" (UID: "adb76aa8-2ad3-4bb6-bea9-ae31de43322e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:08:46.669823 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.669807 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-envoy" pod "adb76aa8-2ad3-4bb6-bea9-ae31de43322e" (UID: "adb76aa8-2ad3-4bb6-bea9-ae31de43322e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:08:46.768736 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.768695 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7684\" (UniqueName: \"kubernetes.io/projected/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-kube-api-access-m7684\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:46.768736 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.768731 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:46.768736 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.768742 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:46.768736 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.768749 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:46.769037 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.768758 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:46.769037 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.768766 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb76aa8-2ad3-4bb6-bea9-ae31de43322e-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:08:46.810359 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.810327 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l"] Oct 30 13:08:46.957165 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:46.957071 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:47.018236 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:47.018200 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40dac033-149e-423e-9630-baeb522263d0" path="/var/lib/kubelet/pods/40dac033-149e-423e-9630-baeb522263d0/volumes" Oct 30 13:08:47.126807 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:47.126774 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:47.404582 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:47.404549 4413 generic.go:356] "Generic (PLEG): container finished" podID="adb76aa8-2ad3-4bb6-bea9-ae31de43322e" containerID="21c846ad397bb2cc4fc7816ee0ece9a035a4200cedd7662c2acbb9dcbd21929f" exitCode=0 Oct 30 13:08:47.404752 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:47.404600 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" Oct 30 13:08:47.404752 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:47.404638 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" event={"ID":"adb76aa8-2ad3-4bb6-bea9-ae31de43322e","Type":"ContainerDied","Data":"21c846ad397bb2cc4fc7816ee0ece9a035a4200cedd7662c2acbb9dcbd21929f"} Oct 30 13:08:47.404752 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:47.404693 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh" event={"ID":"adb76aa8-2ad3-4bb6-bea9-ae31de43322e","Type":"ContainerDied","Data":"8a5125b03ecd40134ecee583df92c25b0e97c95ba93d2ce75a27c9b797af8486"} Oct 30 13:08:47.404752 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:47.404718 4413 scope.go:122] "RemoveContainer" containerID="21c846ad397bb2cc4fc7816ee0ece9a035a4200cedd7662c2acbb9dcbd21929f" Oct 30 13:08:47.414164 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:47.414149 4413 scope.go:122] "RemoveContainer" containerID="926a7ddd3d6509634203cb6dd624ae724b6c25581ef8039181b56296a4d0984b" Oct 30 13:08:47.422357 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:47.422342 4413 scope.go:122] "RemoveContainer" containerID="21c846ad397bb2cc4fc7816ee0ece9a035a4200cedd7662c2acbb9dcbd21929f" Oct 30 13:08:47.422631 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:08:47.422615 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c846ad397bb2cc4fc7816ee0ece9a035a4200cedd7662c2acbb9dcbd21929f\": container with ID starting with 21c846ad397bb2cc4fc7816ee0ece9a035a4200cedd7662c2acbb9dcbd21929f not found: ID does not exist" containerID="21c846ad397bb2cc4fc7816ee0ece9a035a4200cedd7662c2acbb9dcbd21929f" Oct 30 13:08:47.422689 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:47.422640 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c846ad397bb2cc4fc7816ee0ece9a035a4200cedd7662c2acbb9dcbd21929f"} err="failed to get container status \"21c846ad397bb2cc4fc7816ee0ece9a035a4200cedd7662c2acbb9dcbd21929f\": rpc error: code = NotFound desc = could not find container \"21c846ad397bb2cc4fc7816ee0ece9a035a4200cedd7662c2acbb9dcbd21929f\": container with ID starting with 21c846ad397bb2cc4fc7816ee0ece9a035a4200cedd7662c2acbb9dcbd21929f not found: ID does not exist" Oct 30 13:08:47.422689 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:47.422658 4413 scope.go:122] "RemoveContainer" containerID="926a7ddd3d6509634203cb6dd624ae724b6c25581ef8039181b56296a4d0984b" Oct 30 13:08:47.422894 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:08:47.422874 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926a7ddd3d6509634203cb6dd624ae724b6c25581ef8039181b56296a4d0984b\": container with ID starting with 926a7ddd3d6509634203cb6dd624ae724b6c25581ef8039181b56296a4d0984b not found: ID does not exist" containerID="926a7ddd3d6509634203cb6dd624ae724b6c25581ef8039181b56296a4d0984b" Oct 30 13:08:47.422938 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:47.422900 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926a7ddd3d6509634203cb6dd624ae724b6c25581ef8039181b56296a4d0984b"} err="failed to get container status \"926a7ddd3d6509634203cb6dd624ae724b6c25581ef8039181b56296a4d0984b\": rpc error: code = NotFound desc = could not find container \"926a7ddd3d6509634203cb6dd624ae724b6c25581ef8039181b56296a4d0984b\": container with ID starting with 926a7ddd3d6509634203cb6dd624ae724b6c25581ef8039181b56296a4d0984b not found: ID does not exist" Oct 30 13:08:47.466708 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:47.466684 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh"] Oct 30 13:08:47.478465 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:47.478442 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-8d2170-openshift-default-5476557d6f-725gh"] Oct 30 13:08:48.252883 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:48.252853 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:48.381458 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:48.381424 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:48.748138 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:48.748112 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:48.888789 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:48.888761 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:49.018530 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:49.018458 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adb76aa8-2ad3-4bb6-bea9-ae31de43322e" path="/var/lib/kubelet/pods/adb76aa8-2ad3-4bb6-bea9-ae31de43322e/volumes" Oct 30 13:08:50.682643 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:50.682608 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:51.480205 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:51.480179 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:51.602528 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:51.602502 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:52.557289 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:52.557232 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:52.720994 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:52.720953 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:52.945971 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:52.945943 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:53.086684 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:53.086652 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:54.300369 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:54.300338 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:08:54.659646 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:08:54.659615 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:00.496124 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:00.496091 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:01.059149 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:01.059120 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:01.831208 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:01.831162 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" containerID="cri-o://5ab14d6cb78aec997a9e0598a29a67541e76cec490d2fd7a6292a38b4a956c4b" gracePeriod=45 Oct 30 13:09:02.088028 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.087966 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 13:09:02.218730 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.218694 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd22c238-a9e9-4f69-ba78-6d3992925593-kube-api-access-mm25q\" (UniqueName: \"kubernetes.io/projected/cd22c238-a9e9-4f69-ba78-6d3992925593-kube-api-access-mm25q\") pod \"cd22c238-a9e9-4f69-ba78-6d3992925593\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " Oct 30 13:09:02.218905 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.218751 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-tmp-dir\") pod \"cd22c238-a9e9-4f69-ba78-6d3992925593\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " Oct 30 13:09:02.218905 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.218853 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd22c238-a9e9-4f69-ba78-6d3992925593-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd22c238-a9e9-4f69-ba78-6d3992925593-tls-certs\") pod \"cd22c238-a9e9-4f69-ba78-6d3992925593\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " Oct 30 13:09:02.219035 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.218909 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-model-cache\") pod \"cd22c238-a9e9-4f69-ba78-6d3992925593\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " Oct 30 13:09:02.219035 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.218928 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-home\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-home\") pod \"cd22c238-a9e9-4f69-ba78-6d3992925593\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " Oct 30 13:09:02.219035 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.218985 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-dshm\") pod \"cd22c238-a9e9-4f69-ba78-6d3992925593\" (UID: \"cd22c238-a9e9-4f69-ba78-6d3992925593\") " Oct 30 13:09:02.219205 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.219147 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-tmp-dir" pod "cd22c238-a9e9-4f69-ba78-6d3992925593" (UID: "cd22c238-a9e9-4f69-ba78-6d3992925593"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:02.219316 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.219239 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-model-cache" pod "cd22c238-a9e9-4f69-ba78-6d3992925593" (UID: "cd22c238-a9e9-4f69-ba78-6d3992925593"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:02.219369 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.219348 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-home" pod "cd22c238-a9e9-4f69-ba78-6d3992925593" (UID: "cd22c238-a9e9-4f69-ba78-6d3992925593"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:02.219406 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.219382 4413 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-tmp-dir\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.221321 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.221294 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd22c238-a9e9-4f69-ba78-6d3992925593-kube-api-access-mm25q" pod "cd22c238-a9e9-4f69-ba78-6d3992925593" (UID: "cd22c238-a9e9-4f69-ba78-6d3992925593"). InnerVolumeSpecName "kube-api-access-mm25q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:02.221435 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.221317 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd22c238-a9e9-4f69-ba78-6d3992925593-tls-certs" pod "cd22c238-a9e9-4f69-ba78-6d3992925593" (UID: "cd22c238-a9e9-4f69-ba78-6d3992925593"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:02.221435 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.221402 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-dshm" pod "cd22c238-a9e9-4f69-ba78-6d3992925593" (UID: "cd22c238-a9e9-4f69-ba78-6d3992925593"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:09:02.320796 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.320757 4413 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-model-cache\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.320796 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.320791 4413 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-home\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.320796 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.320799 4413 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd22c238-a9e9-4f69-ba78-6d3992925593-dshm\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.321051 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.320809 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm25q\" (UniqueName: \"kubernetes.io/projected/cd22c238-a9e9-4f69-ba78-6d3992925593-kube-api-access-mm25q\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.321051 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.320818 4413 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd22c238-a9e9-4f69-ba78-6d3992925593-tls-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:09:02.492085 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.492047 4413 generic.go:356] "Generic (PLEG): container finished" podID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerID="5ab14d6cb78aec997a9e0598a29a67541e76cec490d2fd7a6292a38b4a956c4b" exitCode=0 Oct 30 13:09:02.492336 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.492101 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" Oct 30 13:09:02.492336 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.492133 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" event={"ID":"cd22c238-a9e9-4f69-ba78-6d3992925593","Type":"ContainerDied","Data":"5ab14d6cb78aec997a9e0598a29a67541e76cec490d2fd7a6292a38b4a956c4b"} Oct 30 13:09:02.492336 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.492183 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l" event={"ID":"cd22c238-a9e9-4f69-ba78-6d3992925593","Type":"ContainerDied","Data":"1e5c8567591788f7735b992d80adc03bb05b33e2bd869295a42176cfe23160d3"} Oct 30 13:09:02.492336 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.492199 4413 scope.go:122] "RemoveContainer" containerID="5ab14d6cb78aec997a9e0598a29a67541e76cec490d2fd7a6292a38b4a956c4b" Oct 30 13:09:02.501456 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.501437 4413 scope.go:122] "RemoveContainer" containerID="5ab14d6cb78aec997a9e0598a29a67541e76cec490d2fd7a6292a38b4a956c4b" Oct 30 13:09:02.501740 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:09:02.501721 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab14d6cb78aec997a9e0598a29a67541e76cec490d2fd7a6292a38b4a956c4b\": container with ID starting with 5ab14d6cb78aec997a9e0598a29a67541e76cec490d2fd7a6292a38b4a956c4b not found: ID does not exist" containerID="5ab14d6cb78aec997a9e0598a29a67541e76cec490d2fd7a6292a38b4a956c4b" Oct 30 13:09:02.501788 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.501748 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab14d6cb78aec997a9e0598a29a67541e76cec490d2fd7a6292a38b4a956c4b"} err="failed to get container status \"5ab14d6cb78aec997a9e0598a29a67541e76cec490d2fd7a6292a38b4a956c4b\": rpc error: code = NotFound desc = could not find container \"5ab14d6cb78aec997a9e0598a29a67541e76cec490d2fd7a6292a38b4a956c4b\": container with ID starting with 5ab14d6cb78aec997a9e0598a29a67541e76cec490d2fd7a6292a38b4a956c4b not found: ID does not exist" Oct 30 13:09:02.542921 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.542888 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l"] Oct 30 13:09:02.554925 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:02.554898 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-512ec5/distinct-2-w4-512ec5-kserve-77f87c6f77-7bb7l"] Oct 30 13:09:03.017612 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:03.017573 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" path="/var/lib/kubelet/pods/cd22c238-a9e9-4f69-ba78-6d3992925593/volumes" Oct 30 13:09:03.555821 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:03.555784 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:03.672179 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:03.672146 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:05.814957 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:09:05.814916 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:09:10.828676 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:10.828630 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598"] Oct 30 13:09:10.829214 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:10.828978 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" podUID="99360f2e-7ffd-4632-9fbb-2289a1ad6b38" containerName="maas-api" containerID="cri-o://e45cede01bf64208ac6d11f9741bcd82a34d09afbbc79f3f6950d462957215d1" gracePeriod=30 Oct 30 13:09:10.903380 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:10.903350 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:11.086897 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.086841 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 13:09:11.209153 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.209126 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-metrics-certs\" (UniqueName: \"kubernetes.io/secret/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-metrics-certs\") pod \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\" (UID: \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\") " Oct 30 13:09:11.209333 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.209236 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-kube-api-access-lsjn6\" (UniqueName: \"kubernetes.io/projected/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-kube-api-access-lsjn6\") pod \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\" (UID: \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\") " Oct 30 13:09:11.209377 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.209329 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-maas-api-tls\") pod \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\" (UID: \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\") " Oct 30 13:09:11.209415 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.209376 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-openshift-service-ca\") pod \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\" (UID: \"99360f2e-7ffd-4632-9fbb-2289a1ad6b38\") " Oct 30 13:09:11.209789 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.209765 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-openshift-service-ca" pod "99360f2e-7ffd-4632-9fbb-2289a1ad6b38" (UID: "99360f2e-7ffd-4632-9fbb-2289a1ad6b38"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:09:11.211594 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.211575 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-maas-api-tls" pod "99360f2e-7ffd-4632-9fbb-2289a1ad6b38" (UID: "99360f2e-7ffd-4632-9fbb-2289a1ad6b38"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:11.211892 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.211869 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-metrics-certs" pod "99360f2e-7ffd-4632-9fbb-2289a1ad6b38" (UID: "99360f2e-7ffd-4632-9fbb-2289a1ad6b38"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 13:09:11.211958 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.211900 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-kube-api-access-lsjn6" pod "99360f2e-7ffd-4632-9fbb-2289a1ad6b38" (UID: "99360f2e-7ffd-4632-9fbb-2289a1ad6b38"). InnerVolumeSpecName "kube-api-access-lsjn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:09:11.310213 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.310179 4413 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-maas-api-tls\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:09:11.310213 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.310208 4413 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-openshift-service-ca\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:09:11.310213 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.310217 4413 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-metrics-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:09:11.310476 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.310226 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsjn6\" (UniqueName: \"kubernetes.io/projected/99360f2e-7ffd-4632-9fbb-2289a1ad6b38-kube-api-access-lsjn6\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:09:11.439626 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.439596 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:11.542091 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.542060 4413 generic.go:356] "Generic (PLEG): container finished" podID="99360f2e-7ffd-4632-9fbb-2289a1ad6b38" containerID="e45cede01bf64208ac6d11f9741bcd82a34d09afbbc79f3f6950d462957215d1" exitCode=0 Oct 30 13:09:11.542278 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.542107 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" Oct 30 13:09:11.542278 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.542108 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" event={"ID":"99360f2e-7ffd-4632-9fbb-2289a1ad6b38","Type":"ContainerDied","Data":"e45cede01bf64208ac6d11f9741bcd82a34d09afbbc79f3f6950d462957215d1"} Oct 30 13:09:11.542278 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.542228 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598" event={"ID":"99360f2e-7ffd-4632-9fbb-2289a1ad6b38","Type":"ContainerDied","Data":"68f64dd35532e7df6eaaedfc6c6a82f3a7f5f07f0398bf793f8564197aff099e"} Oct 30 13:09:11.542278 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.542269 4413 scope.go:122] "RemoveContainer" containerID="e45cede01bf64208ac6d11f9741bcd82a34d09afbbc79f3f6950d462957215d1" Oct 30 13:09:11.552471 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.552452 4413 scope.go:122] "RemoveContainer" containerID="e45cede01bf64208ac6d11f9741bcd82a34d09afbbc79f3f6950d462957215d1" Oct 30 13:09:11.552742 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:09:11.552728 4413 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e45cede01bf64208ac6d11f9741bcd82a34d09afbbc79f3f6950d462957215d1\": container with ID starting with e45cede01bf64208ac6d11f9741bcd82a34d09afbbc79f3f6950d462957215d1 not found: ID does not exist" containerID="e45cede01bf64208ac6d11f9741bcd82a34d09afbbc79f3f6950d462957215d1" Oct 30 13:09:11.552783 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.552750 4413 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e45cede01bf64208ac6d11f9741bcd82a34d09afbbc79f3f6950d462957215d1"} err="failed to get container status \"e45cede01bf64208ac6d11f9741bcd82a34d09afbbc79f3f6950d462957215d1\": rpc error: code = NotFound desc = could not find container \"e45cede01bf64208ac6d11f9741bcd82a34d09afbbc79f3f6950d462957215d1\": container with ID starting with e45cede01bf64208ac6d11f9741bcd82a34d09afbbc79f3f6950d462957215d1 not found: ID does not exist" Oct 30 13:09:11.590027 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.589986 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:11.598324 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.598306 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598"] Oct 30 13:09:11.610286 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:11.610266 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-512ec5-7f8fdbcfbc-qf598"] Oct 30 13:09:12.215447 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:12.215407 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:13.018954 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:13.018925 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99360f2e-7ffd-4632-9fbb-2289a1ad6b38" path="/var/lib/kubelet/pods/99360f2e-7ffd-4632-9fbb-2289a1ad6b38/volumes" Oct 30 13:09:18.544403 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:18.544370 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:19.577242 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:19.577200 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:20.765805 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:20.765774 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-zrh8j_6ec974e2-3c96-493d-b06a-f59ad7d60dce/kuadrant-console-plugin/0.log" Oct 30 13:09:23.810081 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:23.810051 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-g5g58_a63b5bb9-3521-4f1a-bfdd-08325682cfb2/server/0.log" Oct 30 13:09:24.012014 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:24.011962 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mhfr6_7e9919e6-4abf-45ba-bcee-0237e0577a06/manager/0.log" Oct 30 13:09:25.343633 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:25.343595 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:25.465173 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:25.465137 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:25.656914 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:25.656865 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:31.803518 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:31.803488 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:09:35.813608 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:09:35.813560 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:09:37.428215 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:37.428180 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-g5g58_a63b5bb9-3521-4f1a-bfdd-08325682cfb2/server/0.log" Oct 30 13:09:37.642952 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:37.642917 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mhfr6_7e9919e6-4abf-45ba-bcee-0237e0577a06/manager/0.log" Oct 30 13:09:38.917965 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:38.917932 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-57b6f6b66d-kcfsj_bed695ee-4b0b-42cc-9c1f-f658517d9b09/maas-api/0.log" Oct 30 13:09:43.648897 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:43.648871 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-zrh8j_6ec974e2-3c96-493d-b06a-f59ad7d60dce/kuadrant-console-plugin/0.log" Oct 30 13:09:45.905517 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:45.905488 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68fcc45f6c-twgp9_7eea523a-6a90-4f6b-9b89-c6c39bf6513f/kube-auth-proxy/0.log" Oct 30 13:09:46.299953 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:46.299871 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-9ngvj_fe31c94d-c481-44ee-8183-296ce6090146/istio-proxy/0.log" Oct 30 13:09:46.506895 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:46.506863 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-9x4ps_b2d21fd6-1a62-4b69-af88-d5038a533f90/istio-proxy/0.log" Oct 30 13:09:48.454266 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:48.454223 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-wvt6m_b95ec1b9-803b-428c-b6ab-eef315fd03e5/main/0.log" Oct 30 13:09:48.848857 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:48.848754 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gvmqg_d56b1153-6a1c-45c5-bd51-6e8b40ef58ec/main/0.log" Oct 30 13:09:49.052747 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:49.052652 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-l4qzc_da4159b0-d9f6-4cd8-9138-e104f230d251/main/0.log" Oct 30 13:09:49.229820 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:09:49.229792 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-nncf9_be353ed4-7110-45d5-a5f6-489cca57fdcb/main/0.log" Oct 30 13:10:00.292107 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:00.292047 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:00.492191 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:00.492148 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:03.730272 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:03.730226 4413 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" limit=1024 Oct 30 13:10:05.655059 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:05.655022 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj"] Oct 30 13:10:05.655615 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:05.655334 4413 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" podUID="fe31c94d-c481-44ee-8183-296ce6090146" containerName="istio-proxy" containerID="cri-o://5f6adfb5c49ebbafce9010d30fcd3f8c1b4beb05e90c3c69d7fec78eb3e06744" gracePeriod=30 Oct 30 13:10:05.815218 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:10:05.815180 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:10:06.902532 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:06.902496 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pnvz9_68aaec5d-0bc6-4e8b-a311-1797a055bb7b/global-pull-secret-syncer/0.log" Oct 30 13:10:07.484645 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:07.484597 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7lb7d_1a3afa91-aff6-455b-aeda-7275cd8ac8e0/konnectivity-agent/0.log" Oct 30 13:10:08.166094 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:08.166057 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-23.ec2.internal_8314998578de81aba897c7d170293e92/haproxy/0.log" Oct 30 13:10:10.829629 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.829582 4413 generic.go:356] "Generic (PLEG): container finished" podID="fe31c94d-c481-44ee-8183-296ce6090146" containerID="5f6adfb5c49ebbafce9010d30fcd3f8c1b4beb05e90c3c69d7fec78eb3e06744" exitCode=0 Oct 30 13:10:10.830126 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.829666 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" event={"ID":"fe31c94d-c481-44ee-8183-296ce6090146","Type":"ContainerDied","Data":"5f6adfb5c49ebbafce9010d30fcd3f8c1b4beb05e90c3c69d7fec78eb3e06744"} Oct 30 13:10:10.910960 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.910925 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 13:10:10.985274 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.985175 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe31c94d-c481-44ee-8183-296ce6090146-istio-token\" (UniqueName: \"kubernetes.io/projected/fe31c94d-c481-44ee-8183-296ce6090146-istio-token\") pod \"fe31c94d-c481-44ee-8183-296ce6090146\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " Oct 30 13:10:10.985274 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.985225 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-credential-socket\") pod \"fe31c94d-c481-44ee-8183-296ce6090146\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " Oct 30 13:10:10.985530 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.985323 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-istio-data\") pod \"fe31c94d-c481-44ee-8183-296ce6090146\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " Oct 30 13:10:10.985530 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.985359 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-istio-envoy\") pod \"fe31c94d-c481-44ee-8183-296ce6090146\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " Oct 30 13:10:10.985530 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.985395 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-workload-socket\") pod \"fe31c94d-c481-44ee-8183-296ce6090146\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " Oct 30 13:10:10.985530 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.985453 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe31c94d-c481-44ee-8183-296ce6090146-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe31c94d-c481-44ee-8183-296ce6090146-istio-podinfo\") pod \"fe31c94d-c481-44ee-8183-296ce6090146\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " Oct 30 13:10:10.985530 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.985500 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe31c94d-c481-44ee-8183-296ce6090146-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe31c94d-c481-44ee-8183-296ce6090146-istiod-ca-cert\") pod \"fe31c94d-c481-44ee-8183-296ce6090146\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " Oct 30 13:10:10.985821 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.985576 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe31c94d-c481-44ee-8183-296ce6090146-kube-api-access-2rxww\" (UniqueName: \"kubernetes.io/projected/fe31c94d-c481-44ee-8183-296ce6090146-kube-api-access-2rxww\") pod \"fe31c94d-c481-44ee-8183-296ce6090146\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " Oct 30 13:10:10.985821 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.985620 4413 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-workload-certs\") pod \"fe31c94d-c481-44ee-8183-296ce6090146\" (UID: \"fe31c94d-c481-44ee-8183-296ce6090146\") " Oct 30 13:10:10.985821 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.985643 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-credential-socket" pod "fe31c94d-c481-44ee-8183-296ce6090146" (UID: "fe31c94d-c481-44ee-8183-296ce6090146"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:10.985821 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.985779 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-workload-socket" pod "fe31c94d-c481-44ee-8183-296ce6090146" (UID: "fe31c94d-c481-44ee-8183-296ce6090146"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:10.986028 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.985987 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-workload-certs" pod "fe31c94d-c481-44ee-8183-296ce6090146" (UID: "fe31c94d-c481-44ee-8183-296ce6090146"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:10.986087 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.986057 4413 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-credential-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:10:10.986087 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.986080 4413 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-workload-socket\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:10:10.986235 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.986218 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-istio-data" pod "fe31c94d-c481-44ee-8183-296ce6090146" (UID: "fe31c94d-c481-44ee-8183-296ce6090146"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:10.986651 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.986624 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe31c94d-c481-44ee-8183-296ce6090146-istiod-ca-cert" pod "fe31c94d-c481-44ee-8183-296ce6090146" (UID: "fe31c94d-c481-44ee-8183-296ce6090146"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 13:10:10.988570 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.988541 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe31c94d-c481-44ee-8183-296ce6090146-istio-token" pod "fe31c94d-c481-44ee-8183-296ce6090146" (UID: "fe31c94d-c481-44ee-8183-296ce6090146"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:10.988953 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.988930 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe31c94d-c481-44ee-8183-296ce6090146-istio-podinfo" pod "fe31c94d-c481-44ee-8183-296ce6090146" (UID: "fe31c94d-c481-44ee-8183-296ce6090146"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 13:10:10.989450 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.989425 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe31c94d-c481-44ee-8183-296ce6090146-kube-api-access-2rxww" pod "fe31c94d-c481-44ee-8183-296ce6090146" (UID: "fe31c94d-c481-44ee-8183-296ce6090146"). InnerVolumeSpecName "kube-api-access-2rxww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 13:10:10.989676 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:10.989649 4413 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-istio-envoy" pod "fe31c94d-c481-44ee-8183-296ce6090146" (UID: "fe31c94d-c481-44ee-8183-296ce6090146"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 13:10:11.087426 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:11.087382 4413 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe31c94d-c481-44ee-8183-296ce6090146-istio-podinfo\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:10:11.087426 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:11.087416 4413 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe31c94d-c481-44ee-8183-296ce6090146-istiod-ca-cert\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:10:11.087671 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:11.087429 4413 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rxww\" (UniqueName: \"kubernetes.io/projected/fe31c94d-c481-44ee-8183-296ce6090146-kube-api-access-2rxww\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:10:11.087671 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:11.087460 4413 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-workload-certs\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:10:11.087671 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:11.087474 4413 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe31c94d-c481-44ee-8183-296ce6090146-istio-token\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:10:11.087671 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:11.087485 4413 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-istio-data\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:10:11.087671 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:11.087495 4413 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe31c94d-c481-44ee-8183-296ce6090146-istio-envoy\") on node \"ip-10-0-138-23.ec2.internal\" DevicePath \"\"" Oct 30 13:10:11.835904 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:11.835872 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" event={"ID":"fe31c94d-c481-44ee-8183-296ce6090146","Type":"ContainerDied","Data":"793ac5df9b9dc5b97ccc351528593dbef5f793cd5151bd2a9a7656bb46ef248b"} Oct 30 13:10:11.836414 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:11.835932 4413 scope.go:122] "RemoveContainer" containerID="5f6adfb5c49ebbafce9010d30fcd3f8c1b4beb05e90c3c69d7fec78eb3e06744" Oct 30 13:10:11.836414 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:11.835877 4413 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj" Oct 30 13:10:11.909493 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:11.909460 4413 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj"] Oct 30 13:10:11.920791 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:11.920762 4413 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-9ngvj"] Oct 30 13:10:13.019638 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:13.019606 4413 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe31c94d-c481-44ee-8183-296ce6090146" path="/var/lib/kubelet/pods/fe31c94d-c481-44ee-8183-296ce6090146/volumes" Oct 30 13:10:16.606731 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:16.606704 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-zrh8j_6ec974e2-3c96-493d-b06a-f59ad7d60dce/kuadrant-console-plugin/0.log" Oct 30 13:10:18.495305 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495265 4413 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8"] Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495665 4413 cpu_manager.go:414] "containerMap: removing container" podUID="c59d84af-0bbf-4a53-bbe0-07e80541bb86" containerName="payload-pre-processing" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495677 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59d84af-0bbf-4a53-bbe0-07e80541bb86" containerName="payload-pre-processing" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495692 4413 cpu_manager.go:414] "containerMap: removing container" podUID="50eaacfb-1b6b-4cde-854b-95fd32c6dce2" containerName="payload-processing" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495699 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50eaacfb-1b6b-4cde-854b-95fd32c6dce2" containerName="payload-processing" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495709 4413 cpu_manager.go:414] "containerMap: removing container" podUID="75e43f0b-df10-4b4e-b7af-67d515178a84" containerName="istio-proxy" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495714 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e43f0b-df10-4b4e-b7af-67d515178a84" containerName="istio-proxy" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495722 4413 cpu_manager.go:414] "containerMap: removing container" podUID="b770b1fb-b8b3-4f50-93ea-9e88d2e89922" containerName="payload-processing" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495726 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b770b1fb-b8b3-4f50-93ea-9e88d2e89922" containerName="payload-processing" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495733 4413 cpu_manager.go:414] "containerMap: removing container" podUID="0e06b3dc-2413-4f9f-ba0d-691519d450f1" containerName="istio-proxy" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495738 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e06b3dc-2413-4f9f-ba0d-691519d450f1" containerName="istio-proxy" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495744 4413 cpu_manager.go:414] "containerMap: removing container" podUID="adb76aa8-2ad3-4bb6-bea9-ae31de43322e" containerName="istio-proxy" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495748 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb76aa8-2ad3-4bb6-bea9-ae31de43322e" containerName="istio-proxy" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495755 4413 cpu_manager.go:414] "containerMap: removing container" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495760 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495769 4413 cpu_manager.go:414] "containerMap: removing container" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerName="main" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495774 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerName="main" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495781 4413 cpu_manager.go:414] "containerMap: removing container" podUID="ed73650f-c307-4db1-80c1-9b129dd488c4" containerName="payload-pre-processing" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495786 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed73650f-c307-4db1-80c1-9b129dd488c4" containerName="payload-pre-processing" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495793 4413 cpu_manager.go:414] "containerMap: removing container" podUID="951adfc0-673a-4716-b07a-e94182a7e608" containerName="maas-api" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495798 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951adfc0-673a-4716-b07a-e94182a7e608" containerName="maas-api" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495804 4413 cpu_manager.go:414] "containerMap: removing container" podUID="935f35f7-28c0-4d05-ab8e-1a2042d7aab4" containerName="payload-pre-processing" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495809 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935f35f7-28c0-4d05-ab8e-1a2042d7aab4" containerName="payload-pre-processing" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495818 4413 cpu_manager.go:414] "containerMap: removing container" podUID="52a0865a-7865-4813-a6a9-05ff39054503" containerName="cleanup" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495822 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a0865a-7865-4813-a6a9-05ff39054503" containerName="cleanup" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495828 4413 cpu_manager.go:414] "containerMap: removing container" podUID="5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b" containerName="cleanup" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495832 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b" containerName="cleanup" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495839 4413 cpu_manager.go:414] "containerMap: removing container" podUID="2232e7b3-95ec-4866-81a9-30a396bf9d96" containerName="payload-processing" Oct 30 13:10:18.495830 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495843 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2232e7b3-95ec-4866-81a9-30a396bf9d96" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495851 4413 cpu_manager.go:414] "containerMap: removing container" podUID="0c2f60ee-cdf2-4a48-a6ff-092f61a349e2" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495855 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2f60ee-cdf2-4a48-a6ff-092f61a349e2" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495863 4413 cpu_manager.go:414] "containerMap: removing container" podUID="99b68128-c265-4be3-a666-d3e73ad7fa39" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495867 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99b68128-c265-4be3-a666-d3e73ad7fa39" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495873 4413 cpu_manager.go:414] "containerMap: removing container" podUID="0e06b3dc-2413-4f9f-ba0d-691519d450f1" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495877 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e06b3dc-2413-4f9f-ba0d-691519d450f1" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495883 4413 cpu_manager.go:414] "containerMap: removing container" podUID="4c30ec11-6046-4da2-944a-c45f40848016" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495887 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c30ec11-6046-4da2-944a-c45f40848016" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495895 4413 cpu_manager.go:414] "containerMap: removing container" podUID="fdabf3ad-4ef3-4f8e-9f53-6565dacf505e" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495900 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdabf3ad-4ef3-4f8e-9f53-6565dacf505e" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495908 4413 cpu_manager.go:414] "containerMap: removing container" podUID="0818b575-9f29-4241-967b-448947aaf982" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495912 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0818b575-9f29-4241-967b-448947aaf982" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495919 4413 cpu_manager.go:414] "containerMap: removing container" podUID="8c843f24-0220-4063-8d6d-d1f84196d9b2" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495923 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c843f24-0220-4063-8d6d-d1f84196d9b2" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495930 4413 cpu_manager.go:414] "containerMap: removing container" podUID="ad21c9c0-28e9-4e5c-b82b-64747d88bc93" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495935 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad21c9c0-28e9-4e5c-b82b-64747d88bc93" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495942 4413 cpu_manager.go:414] "containerMap: removing container" podUID="27ef9a37-c090-4cbb-824f-f0a9c54d3fe2" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495946 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ef9a37-c090-4cbb-824f-f0a9c54d3fe2" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495954 4413 cpu_manager.go:414] "containerMap: removing container" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495957 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495963 4413 cpu_manager.go:414] "containerMap: removing container" podUID="1dde17ed-ff85-43dd-8cd7-a702a97bbd3b" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495967 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dde17ed-ff85-43dd-8cd7-a702a97bbd3b" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495974 4413 cpu_manager.go:414] "containerMap: removing container" podUID="adb76aa8-2ad3-4bb6-bea9-ae31de43322e" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495978 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb76aa8-2ad3-4bb6-bea9-ae31de43322e" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495986 4413 cpu_manager.go:414] "containerMap: removing container" podUID="74ccd1d2-3901-46f4-b765-2ace6683fad1" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495990 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ccd1d2-3901-46f4-b765-2ace6683fad1" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.495996 4413 cpu_manager.go:414] "containerMap: removing container" podUID="fd580504-41ed-4b72-b393-62d3cc8a49b2" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496000 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd580504-41ed-4b72-b393-62d3cc8a49b2" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496006 4413 cpu_manager.go:414] "containerMap: removing container" podUID="4aa9879d-dab4-42ea-8dd8-ad6bba64214c" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496011 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa9879d-dab4-42ea-8dd8-ad6bba64214c" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496018 4413 cpu_manager.go:414] "containerMap: removing container" podUID="ee65528d-b306-4f62-97fe-6314d1d9660e" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496022 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee65528d-b306-4f62-97fe-6314d1d9660e" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496028 4413 cpu_manager.go:414] "containerMap: removing container" podUID="835438be-d2d1-40da-a7ed-cd420db69319" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496033 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835438be-d2d1-40da-a7ed-cd420db69319" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496041 4413 cpu_manager.go:414] "containerMap: removing container" podUID="5b66ed77-d0a7-4180-ba18-8598280715af" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496045 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b66ed77-d0a7-4180-ba18-8598280715af" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496051 4413 cpu_manager.go:414] "containerMap: removing container" podUID="52a0865a-7865-4813-a6a9-05ff39054503" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496055 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a0865a-7865-4813-a6a9-05ff39054503" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496063 4413 cpu_manager.go:414] "containerMap: removing container" podUID="fe249b4d-be8a-4135-acfc-568903715ef6" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496067 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe249b4d-be8a-4135-acfc-568903715ef6" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496073 4413 cpu_manager.go:414] "containerMap: removing container" podUID="6435f44e-980e-4166-868b-ff94fa2010b9" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496078 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6435f44e-980e-4166-868b-ff94fa2010b9" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496084 4413 cpu_manager.go:414] "containerMap: removing container" podUID="5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496088 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496097 4413 cpu_manager.go:414] "containerMap: removing container" podUID="272a58bb-d7d5-4c02-a278-1dd57cdc75bf" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496101 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="272a58bb-d7d5-4c02-a278-1dd57cdc75bf" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496109 4413 cpu_manager.go:414] "containerMap: removing container" podUID="6435f44e-980e-4166-868b-ff94fa2010b9" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496113 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6435f44e-980e-4166-868b-ff94fa2010b9" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496120 4413 cpu_manager.go:414] "containerMap: removing container" podUID="d94ea46b-0895-4536-a9c6-4088e1116400" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496123 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d94ea46b-0895-4536-a9c6-4088e1116400" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496130 4413 cpu_manager.go:414] "containerMap: removing container" podUID="747e931b-c949-4428-afe2-16da8de97a9a" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496134 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747e931b-c949-4428-afe2-16da8de97a9a" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496140 4413 cpu_manager.go:414] "containerMap: removing container" podUID="fe31c94d-c481-44ee-8183-296ce6090146" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496144 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe31c94d-c481-44ee-8183-296ce6090146" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496150 4413 cpu_manager.go:414] "containerMap: removing container" podUID="8133b92d-e5ca-4a5f-94bc-9ee436f87cfc" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496154 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8133b92d-e5ca-4a5f-94bc-9ee436f87cfc" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496160 4413 cpu_manager.go:414] "containerMap: removing container" podUID="9c32f618-5687-4d8a-bd81-05e3881d92e0" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496164 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c32f618-5687-4d8a-bd81-05e3881d92e0" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496171 4413 cpu_manager.go:414] "containerMap: removing container" podUID="22d9217c-a901-46fa-ba61-21bc01e658cb" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496176 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d9217c-a901-46fa-ba61-21bc01e658cb" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496181 4413 cpu_manager.go:414] "containerMap: removing container" podUID="83528507-6527-4816-a0b7-abdf57f86dd9" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496186 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83528507-6527-4816-a0b7-abdf57f86dd9" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496192 4413 cpu_manager.go:414] "containerMap: removing container" podUID="83528507-6527-4816-a0b7-abdf57f86dd9" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496196 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83528507-6527-4816-a0b7-abdf57f86dd9" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496203 4413 cpu_manager.go:414] "containerMap: removing container" podUID="d7a63529-8b71-40df-b440-d8b5316d112a" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496208 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a63529-8b71-40df-b440-d8b5316d112a" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496216 4413 cpu_manager.go:414] "containerMap: removing container" podUID="fd2d9a96-1d2e-4eee-946a-d26618a955ca" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496220 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2d9a96-1d2e-4eee-946a-d26618a955ca" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496227 4413 cpu_manager.go:414] "containerMap: removing container" podUID="ae223397-5523-4836-9d6c-7e2daf2ae313" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496231 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae223397-5523-4836-9d6c-7e2daf2ae313" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496240 4413 cpu_manager.go:414] "containerMap: removing container" podUID="68a11193-ed13-452f-b0b1-eaeb60b21465" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496258 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68a11193-ed13-452f-b0b1-eaeb60b21465" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496267 4413 cpu_manager.go:414] "containerMap: removing container" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496272 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496277 4413 cpu_manager.go:414] "containerMap: removing container" podUID="25322f93-360d-472f-8c1b-382c04505663" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496282 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25322f93-360d-472f-8c1b-382c04505663" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496289 4413 cpu_manager.go:414] "containerMap: removing container" podUID="37eb395b-d510-4d1c-a59c-50d5d42b0efd" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496293 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37eb395b-d510-4d1c-a59c-50d5d42b0efd" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496298 4413 cpu_manager.go:414] "containerMap: removing container" podUID="b8fc0118-77f2-457b-8a58-82882ee852d0" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496302 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8fc0118-77f2-457b-8a58-82882ee852d0" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496308 4413 cpu_manager.go:414] "containerMap: removing container" podUID="a8038e2f-a57c-490f-9206-84b5c379830a" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496311 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8038e2f-a57c-490f-9206-84b5c379830a" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496318 4413 cpu_manager.go:414] "containerMap: removing container" podUID="ea820a52-0cbd-4fc8-9539-4aa9810e1836" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496323 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea820a52-0cbd-4fc8-9539-4aa9810e1836" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496330 4413 cpu_manager.go:414] "containerMap: removing container" podUID="99360f2e-7ffd-4632-9fbb-2289a1ad6b38" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496334 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99360f2e-7ffd-4632-9fbb-2289a1ad6b38" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496340 4413 cpu_manager.go:414] "containerMap: removing container" podUID="a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496344 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496349 4413 cpu_manager.go:414] "containerMap: removing container" podUID="8133b92d-e5ca-4a5f-94bc-9ee436f87cfc" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496355 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8133b92d-e5ca-4a5f-94bc-9ee436f87cfc" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496363 4413 cpu_manager.go:414] "containerMap: removing container" podUID="70146efe-2585-4812-8761-50cd5c027a1c" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496367 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70146efe-2585-4812-8761-50cd5c027a1c" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496375 4413 cpu_manager.go:414] "containerMap: removing container" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496380 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496387 4413 cpu_manager.go:414] "containerMap: removing container" podUID="26a8eae2-4b05-4be5-9231-548a3b651406" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496391 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a8eae2-4b05-4be5-9231-548a3b651406" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496396 4413 cpu_manager.go:414] "containerMap: removing container" podUID="8133b92d-e5ca-4a5f-94bc-9ee436f87cfc" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496400 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8133b92d-e5ca-4a5f-94bc-9ee436f87cfc" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496408 4413 cpu_manager.go:414] "containerMap: removing container" podUID="40dac033-149e-423e-9630-baeb522263d0" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496412 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40dac033-149e-423e-9630-baeb522263d0" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496418 4413 cpu_manager.go:414] "containerMap: removing container" podUID="18816891-50b2-484b-bdb7-ecb40f57097f" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496422 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18816891-50b2-484b-bdb7-ecb40f57097f" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496428 4413 cpu_manager.go:414] "containerMap: removing container" podUID="a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496432 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496438 4413 cpu_manager.go:414] "containerMap: removing container" podUID="0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496442 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496448 4413 cpu_manager.go:414] "containerMap: removing container" podUID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496455 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496461 4413 cpu_manager.go:414] "containerMap: removing container" podUID="4295a82c-12f5-4451-b82a-0203044f1de3" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496465 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4295a82c-12f5-4451-b82a-0203044f1de3" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496471 4413 cpu_manager.go:414] "containerMap: removing container" podUID="4fb8871c-9f64-44a2-bc8e-c110b777d677" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496475 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fb8871c-9f64-44a2-bc8e-c110b777d677" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496483 4413 cpu_manager.go:414] "containerMap: removing container" podUID="f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496487 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496493 4413 cpu_manager.go:414] "containerMap: removing container" podUID="9c32f618-5687-4d8a-bd81-05e3881d92e0" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496497 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c32f618-5687-4d8a-bd81-05e3881d92e0" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496505 4413 cpu_manager.go:414] "containerMap: removing container" podUID="e2971b88-5b87-49db-b7f8-60c2a9d7116d" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496509 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2971b88-5b87-49db-b7f8-60c2a9d7116d" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496515 4413 cpu_manager.go:414] "containerMap: removing container" podUID="29b11889-7268-40a1-8387-109af1464e04" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496520 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b11889-7268-40a1-8387-109af1464e04" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496526 4413 cpu_manager.go:414] "containerMap: removing container" podUID="a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496530 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496537 4413 cpu_manager.go:414] "containerMap: removing container" podUID="f9b9b231-747e-45be-8024-7f1b05facd48" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496542 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b9b231-747e-45be-8024-7f1b05facd48" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496548 4413 cpu_manager.go:414] "containerMap: removing container" podUID="d3832da5-41b0-4993-b87d-971b41984eb3" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496552 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3832da5-41b0-4993-b87d-971b41984eb3" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496559 4413 cpu_manager.go:414] "containerMap: removing container" podUID="18816891-50b2-484b-bdb7-ecb40f57097f" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496563 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18816891-50b2-484b-bdb7-ecb40f57097f" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496571 4413 cpu_manager.go:414] "containerMap: removing container" podUID="5bfd491e-874f-4edd-b01b-b5bc8d20278b" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496574 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfd491e-874f-4edd-b01b-b5bc8d20278b" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496579 4413 cpu_manager.go:414] "containerMap: removing container" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496583 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496589 4413 cpu_manager.go:414] "containerMap: removing container" podUID="9aa641d6-2556-4e48-9503-85d33c43df8a" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496593 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aa641d6-2556-4e48-9503-85d33c43df8a" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496598 4413 cpu_manager.go:414] "containerMap: removing container" podUID="ac48ad61-b11f-46c7-902e-87fa0d893933" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496603 4413 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac48ad61-b11f-46c7-902e-87fa0d893933" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496662 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="37eb395b-d510-4d1c-a59c-50d5d42b0efd" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496671 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8fc0118-77f2-457b-8a58-82882ee852d0" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496677 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8038e2f-a57c-490f-9206-84b5c379830a" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496682 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c32f618-5687-4d8a-bd81-05e3881d92e0" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496690 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea820a52-0cbd-4fc8-9539-4aa9810e1836" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496695 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="83528507-6527-4816-a0b7-abdf57f86dd9" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496702 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="99360f2e-7ffd-4632-9fbb-2289a1ad6b38" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496708 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7cb8bea-e8fb-4ff2-8d0d-2e6cf8a46da7" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496713 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="8133b92d-e5ca-4a5f-94bc-9ee436f87cfc" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496718 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="70146efe-2585-4812-8761-50cd5c027a1c" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496722 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a0865a-7865-4813-a6a9-05ff39054503" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496727 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="93cdeb23-4ab6-4708-a0a2-e1bfaac5e3cf" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496733 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a8eae2-4b05-4be5-9231-548a3b651406" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496738 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="8133b92d-e5ca-4a5f-94bc-9ee436f87cfc" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496744 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="40dac033-149e-423e-9630-baeb522263d0" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496749 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="18816891-50b2-484b-bdb7-ecb40f57097f" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496755 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e43f0b-df10-4b4e-b7af-67d515178a84" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496759 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="0198e9a4-ced1-4c15-a6d0-9c9ef3ea1c40" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496765 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ea2a69-24a6-4f0b-adb0-06a662f793fa" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496770 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="4295a82c-12f5-4451-b82a-0203044f1de3" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496776 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fb8871c-9f64-44a2-bc8e-c110b777d677" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496782 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="f26b0f59-0bb1-4015-8c3b-196a6f8f8cd2" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496787 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c32f618-5687-4d8a-bd81-05e3881d92e0" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496791 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="18816891-50b2-484b-bdb7-ecb40f57097f" containerName="cleanup" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496795 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2971b88-5b87-49db-b7f8-60c2a9d7116d" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496801 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b11889-7268-40a1-8387-109af1464e04" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496807 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5fd2f7f-f7b7-44cb-9993-df17cfb6cd39" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496811 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b9b231-747e-45be-8024-7f1b05facd48" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496817 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b66ed77-d0a7-4180-ba18-8598280715af" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496821 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3832da5-41b0-4993-b87d-971b41984eb3" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496828 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfd491e-874f-4edd-b01b-b5bc8d20278b" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496833 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd22c238-a9e9-4f69-ba78-6d3992925593" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496838 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aa641d6-2556-4e48-9503-85d33c43df8a" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496842 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac48ad61-b11f-46c7-902e-87fa0d893933" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496846 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59d84af-0bbf-4a53-bbe0-07e80541bb86" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496851 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="50eaacfb-1b6b-4cde-854b-95fd32c6dce2" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496856 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e43f0b-df10-4b4e-b7af-67d515178a84" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496862 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="b770b1fb-b8b3-4f50-93ea-9e88d2e89922" containerName="payload-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496866 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e06b3dc-2413-4f9f-ba0d-691519d450f1" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496870 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb76aa8-2ad3-4bb6-bea9-ae31de43322e" containerName="istio-proxy" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496877 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c57b1c-6aec-48f0-81aa-adac4196039e" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496882 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="85c3372a-f4c6-450f-8de9-5a826d3ee1f6" containerName="main" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496888 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed73650f-c307-4db1-80c1-9b129dd488c4" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496893 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="951adfc0-673a-4716-b07a-e94182a7e608" containerName="maas-api" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496897 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="935f35f7-28c0-4d05-ab8e-1a2042d7aab4" containerName="payload-pre-processing" Oct 30 13:10:18.496916 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496902 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a0865a-7865-4813-a6a9-05ff39054503" containerName="cleanup" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496906 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b" containerName="cleanup" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496910 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="2232e7b3-95ec-4866-81a9-30a396bf9d96" containerName="payload-processing" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496915 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2f60ee-cdf2-4a48-a6ff-092f61a349e2" containerName="payload-processing" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496919 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="99b68128-c265-4be3-a666-d3e73ad7fa39" containerName="payload-processing" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496925 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c30ec11-6046-4da2-944a-c45f40848016" containerName="maas-api" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496930 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdabf3ad-4ef3-4f8e-9f53-6565dacf505e" containerName="payload-pre-processing" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496935 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="0818b575-9f29-4241-967b-448947aaf982" containerName="payload-processing" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496940 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c843f24-0220-4063-8d6d-d1f84196d9b2" containerName="maas-api" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496946 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad21c9c0-28e9-4e5c-b82b-64747d88bc93" containerName="payload-pre-processing" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496951 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ef9a37-c090-4cbb-824f-f0a9c54d3fe2" containerName="maas-api" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496956 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdebed52-2014-447e-975a-3b6c02c13535" containerName="main" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496960 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dde17ed-ff85-43dd-8cd7-a702a97bbd3b" containerName="payload-pre-processing" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496966 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb76aa8-2ad3-4bb6-bea9-ae31de43322e" containerName="istio-proxy" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496972 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ccd1d2-3901-46f4-b765-2ace6683fad1" containerName="maas-api" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496978 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd580504-41ed-4b72-b393-62d3cc8a49b2" containerName="istio-proxy" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496982 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa9879d-dab4-42ea-8dd8-ad6bba64214c" containerName="maas-api" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496987 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee65528d-b306-4f62-97fe-6314d1d9660e" containerName="maas-api" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496992 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="835438be-d2d1-40da-a7ed-cd420db69319" containerName="payload-processing" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.496997 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b66ed77-d0a7-4180-ba18-8598280715af" containerName="istio-proxy" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.497002 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a0865a-7865-4813-a6a9-05ff39054503" containerName="cleanup" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.497006 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe249b4d-be8a-4135-acfc-568903715ef6" containerName="payload-processing" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.497012 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="6435f44e-980e-4166-868b-ff94fa2010b9" containerName="istio-proxy" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.497016 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d45cce5-ffa8-41cc-b1cb-136b43d3bc9b" containerName="cleanup" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.497021 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="272a58bb-d7d5-4c02-a278-1dd57cdc75bf" containerName="payload-pre-processing" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.497027 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="6435f44e-980e-4166-868b-ff94fa2010b9" containerName="istio-proxy" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.497032 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="d94ea46b-0895-4536-a9c6-4088e1116400" containerName="maas-api" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.497037 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="747e931b-c949-4428-afe2-16da8de97a9a" containerName="maas-api" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.497043 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe31c94d-c481-44ee-8183-296ce6090146" containerName="istio-proxy" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.497047 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="8133b92d-e5ca-4a5f-94bc-9ee436f87cfc" containerName="cleanup" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.497052 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d9217c-a901-46fa-ba61-21bc01e658cb" containerName="maas-api" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.497056 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="83528507-6527-4816-a0b7-abdf57f86dd9" containerName="cleanup" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.497062 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a63529-8b71-40df-b440-d8b5316d112a" containerName="payload-pre-processing" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.497067 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2d9a96-1d2e-4eee-946a-d26618a955ca" containerName="maas-api" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.497072 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae223397-5523-4836-9d6c-7e2daf2ae313" containerName="payload-pre-processing" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.497077 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="68a11193-ed13-452f-b0b1-eaeb60b21465" containerName="payload-processing" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.497083 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d24c3bd-af3d-4379-a8bf-7f40a42b6e90" containerName="main" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.497088 4413 memory_manager.go:365] "RemoveStaleState removing state" podUID="25322f93-360d-472f-8c1b-382c04505663" containerName="payload-processing" Oct 30 13:10:18.502859 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.500092 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:18.518103 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.518070 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8"] Oct 30 13:10:18.528060 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.528033 4413 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-47mpn\"/\"default-dockercfg-xw6qs\"" Oct 30 13:10:18.568925 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.561728 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47mpn\"/\"openshift-service-ca.crt\"" Oct 30 13:10:18.568925 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.561858 4413 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47mpn\"/\"kube-root-ca.crt\"" Oct 30 13:10:18.664268 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.664216 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38811999-5fc2-462a-a57f-65bfc0bffb80-sys\") pod \"perf-node-gather-daemonset-5dkw8\" (UID: \"38811999-5fc2-462a-a57f-65bfc0bffb80\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:18.664434 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.664353 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38811999-5fc2-462a-a57f-65bfc0bffb80-proc\") pod \"perf-node-gather-daemonset-5dkw8\" (UID: \"38811999-5fc2-462a-a57f-65bfc0bffb80\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:18.664522 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.664485 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38811999-5fc2-462a-a57f-65bfc0bffb80-podres\") pod \"perf-node-gather-daemonset-5dkw8\" (UID: \"38811999-5fc2-462a-a57f-65bfc0bffb80\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:18.664587 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.664529 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38811999-5fc2-462a-a57f-65bfc0bffb80-lib-modules\") pod \"perf-node-gather-daemonset-5dkw8\" (UID: \"38811999-5fc2-462a-a57f-65bfc0bffb80\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:18.664652 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.664628 4413 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jq8l\" (UniqueName: \"kubernetes.io/projected/38811999-5fc2-462a-a57f-65bfc0bffb80-kube-api-access-9jq8l\") pod \"perf-node-gather-daemonset-5dkw8\" (UID: \"38811999-5fc2-462a-a57f-65bfc0bffb80\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:18.766014 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.765910 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38811999-5fc2-462a-a57f-65bfc0bffb80-proc\") pod \"perf-node-gather-daemonset-5dkw8\" (UID: \"38811999-5fc2-462a-a57f-65bfc0bffb80\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:18.766014 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.765998 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38811999-5fc2-462a-a57f-65bfc0bffb80-podres\") pod \"perf-node-gather-daemonset-5dkw8\" (UID: \"38811999-5fc2-462a-a57f-65bfc0bffb80\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:18.766282 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.766031 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38811999-5fc2-462a-a57f-65bfc0bffb80-lib-modules\") pod \"perf-node-gather-daemonset-5dkw8\" (UID: \"38811999-5fc2-462a-a57f-65bfc0bffb80\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:18.766282 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.766056 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38811999-5fc2-462a-a57f-65bfc0bffb80-proc\") pod \"perf-node-gather-daemonset-5dkw8\" (UID: \"38811999-5fc2-462a-a57f-65bfc0bffb80\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:18.766282 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.766073 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jq8l\" (UniqueName: \"kubernetes.io/projected/38811999-5fc2-462a-a57f-65bfc0bffb80-kube-api-access-9jq8l\") pod \"perf-node-gather-daemonset-5dkw8\" (UID: \"38811999-5fc2-462a-a57f-65bfc0bffb80\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:18.766282 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.766126 4413 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38811999-5fc2-462a-a57f-65bfc0bffb80-sys\") pod \"perf-node-gather-daemonset-5dkw8\" (UID: \"38811999-5fc2-462a-a57f-65bfc0bffb80\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:18.766282 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.766142 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38811999-5fc2-462a-a57f-65bfc0bffb80-podres\") pod \"perf-node-gather-daemonset-5dkw8\" (UID: \"38811999-5fc2-462a-a57f-65bfc0bffb80\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:18.766282 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.766219 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38811999-5fc2-462a-a57f-65bfc0bffb80-sys\") pod \"perf-node-gather-daemonset-5dkw8\" (UID: \"38811999-5fc2-462a-a57f-65bfc0bffb80\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:18.766282 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.766264 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38811999-5fc2-462a-a57f-65bfc0bffb80-lib-modules\") pod \"perf-node-gather-daemonset-5dkw8\" (UID: \"38811999-5fc2-462a-a57f-65bfc0bffb80\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:18.789720 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.789688 4413 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jq8l\" (UniqueName: \"kubernetes.io/projected/38811999-5fc2-462a-a57f-65bfc0bffb80-kube-api-access-9jq8l\") pod \"perf-node-gather-daemonset-5dkw8\" (UID: \"38811999-5fc2-462a-a57f-65bfc0bffb80\") " pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:18.813043 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:18.813010 4413 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:19.112193 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:19.112160 4413 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8"] Oct 30 13:10:19.115122 ip-10-0-138-23 kubenswrapper[4413]: W1030 13:10:19.115088 4413 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod38811999_5fc2_462a_a57f_65bfc0bffb80.slice/crio-f84c6a4882c87f32cf27aa4266e0e145a9b2cbc6e20f653f10bac9065effb85d WatchSource:0}: Error finding container f84c6a4882c87f32cf27aa4266e0e145a9b2cbc6e20f653f10bac9065effb85d: Status 404 returned error can't find the container with id f84c6a4882c87f32cf27aa4266e0e145a9b2cbc6e20f653f10bac9065effb85d Oct 30 13:10:19.519847 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:19.519824 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-g5g58_a63b5bb9-3521-4f1a-bfdd-08325682cfb2/server/0.log" Oct 30 13:10:19.523621 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:19.523598 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_148b3790-0c62-4d7f-be84-be2bdc04595c/alertmanager/0.log" Oct 30 13:10:19.678198 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:19.678172 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_148b3790-0c62-4d7f-be84-be2bdc04595c/config-reloader/0.log" Oct 30 13:10:19.824279 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:19.824175 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_148b3790-0c62-4d7f-be84-be2bdc04595c/kube-rbac-proxy-web/0.log" Oct 30 13:10:19.877784 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:19.877753 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" event={"ID":"38811999-5fc2-462a-a57f-65bfc0bffb80","Type":"ContainerStarted","Data":"e0f403367d36af2b363e6dbf834bd691afb1fd01147c20244ea2b8b8094ad5e9"} Oct 30 13:10:19.878005 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:19.877809 4413 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" event={"ID":"38811999-5fc2-462a-a57f-65bfc0bffb80","Type":"ContainerStarted","Data":"f84c6a4882c87f32cf27aa4266e0e145a9b2cbc6e20f653f10bac9065effb85d"} Oct 30 13:10:19.878005 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:19.877856 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:19.920840 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:19.920785 4413 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" podStartSLOduration=1.920758755 podStartE2EDuration="1.920758755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:19.918456041 +0000 UTC m=+4573.028298922" watchObservedRunningTime="2026-10-30 13:10:19.920758755 +0000 UTC m=+4573.030601633" Oct 30 13:10:19.960025 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:19.959980 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_148b3790-0c62-4d7f-be84-be2bdc04595c/kube-rbac-proxy/0.log" Oct 30 13:10:20.252067 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:20.252035 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_148b3790-0c62-4d7f-be84-be2bdc04595c/kube-rbac-proxy-metric/0.log" Oct 30 13:10:20.391075 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:20.391043 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_148b3790-0c62-4d7f-be84-be2bdc04595c/prom-label-proxy/0.log" Oct 30 13:10:20.529002 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:20.528884 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_148b3790-0c62-4d7f-be84-be2bdc04595c/init-config-reloader/0.log" Oct 30 13:10:20.803370 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:20.803287 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-9q6m9_ea0bc518-3942-4980-8d19-dfd59954169a/kube-state-metrics/0.log" Oct 30 13:10:20.914937 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:20.914912 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-9q6m9_ea0bc518-3942-4980-8d19-dfd59954169a/kube-rbac-proxy-main/0.log" Oct 30 13:10:20.948482 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:20.948451 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-g5g58_a63b5bb9-3521-4f1a-bfdd-08325682cfb2/server/0.log" Oct 30 13:10:21.085941 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:21.085850 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-9q6m9_ea0bc518-3942-4980-8d19-dfd59954169a/kube-rbac-proxy-self/0.log" Oct 30 13:10:21.100694 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:21.100662 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mhfr6_7e9919e6-4abf-45ba-bcee-0237e0577a06/manager/0.log" Oct 30 13:10:21.291476 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:21.291447 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-94cc96cf4-mq2jm_f122aa99-4d33-450f-acd4-da132dd1cb2e/metrics-server/0.log" Oct 30 13:10:21.423999 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:21.423967 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-rqhdc_59d12821-d129-4883-89c6-219a359e3143/monitoring-plugin/0.log" Oct 30 13:10:21.549385 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:21.549349 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bgk7n_18e30a28-a964-429e-8305-f5ffbe40ba3b/node-exporter/0.log" Oct 30 13:10:21.666298 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:21.666260 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bgk7n_18e30a28-a964-429e-8305-f5ffbe40ba3b/kube-rbac-proxy/0.log" Oct 30 13:10:21.786752 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:21.786662 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bgk7n_18e30a28-a964-429e-8305-f5ffbe40ba3b/init-textfile/0.log" Oct 30 13:10:22.394268 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:22.394218 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mhfr6_7e9919e6-4abf-45ba-bcee-0237e0577a06/manager/0.log" Oct 30 13:10:23.070432 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:23.070401 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06a5bd76-a68f-4c5a-a085-d3f1d91724d5/prometheus/0.log" Oct 30 13:10:23.190108 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:23.190035 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06a5bd76-a68f-4c5a-a085-d3f1d91724d5/config-reloader/0.log" Oct 30 13:10:23.313492 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:23.313459 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06a5bd76-a68f-4c5a-a085-d3f1d91724d5/thanos-sidecar/0.log" Oct 30 13:10:23.590964 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:23.590889 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06a5bd76-a68f-4c5a-a085-d3f1d91724d5/kube-rbac-proxy-web/0.log" Oct 30 13:10:23.717976 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:23.717940 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06a5bd76-a68f-4c5a-a085-d3f1d91724d5/kube-rbac-proxy/0.log" Oct 30 13:10:23.839928 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:23.839899 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06a5bd76-a68f-4c5a-a085-d3f1d91724d5/kube-rbac-proxy-thanos/0.log" Oct 30 13:10:23.962217 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:23.962162 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_06a5bd76-a68f-4c5a-a085-d3f1d91724d5/init-config-reloader/0.log" Oct 30 13:10:24.028709 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:24.028679 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-g5g58_a63b5bb9-3521-4f1a-bfdd-08325682cfb2/server/0.log" Oct 30 13:10:24.512867 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:24.512830 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65459c448-trpdm_2f8e0b61-2169-4d89-8315-4092541ac795/telemeter-client/0.log" Oct 30 13:10:24.652274 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:24.652218 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65459c448-trpdm_2f8e0b61-2169-4d89-8315-4092541ac795/reload/0.log" Oct 30 13:10:24.816585 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:24.816490 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65459c448-trpdm_2f8e0b61-2169-4d89-8315-4092541ac795/kube-rbac-proxy/0.log" Oct 30 13:10:25.002029 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:25.001989 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8676c985b4-fm9dg_c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1/thanos-query/0.log" Oct 30 13:10:25.117819 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:25.117743 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8676c985b4-fm9dg_c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1/kube-rbac-proxy-web/0.log" Oct 30 13:10:25.237780 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:25.237758 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8676c985b4-fm9dg_c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1/kube-rbac-proxy/0.log" Oct 30 13:10:25.362372 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:25.362340 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8676c985b4-fm9dg_c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1/prom-label-proxy/0.log" Oct 30 13:10:25.459794 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:25.459747 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mhfr6_7e9919e6-4abf-45ba-bcee-0237e0577a06/manager/0.log" Oct 30 13:10:25.475498 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:25.475470 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-g5g58_a63b5bb9-3521-4f1a-bfdd-08325682cfb2/server/0.log" Oct 30 13:10:25.478585 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:25.478555 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8676c985b4-fm9dg_c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1/kube-rbac-proxy-rules/0.log" Oct 30 13:10:25.599390 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:25.599360 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8676c985b4-fm9dg_c0d0a2b0-8520-448b-aa9f-cd4e43f84ff1/kube-rbac-proxy-metrics/0.log" Oct 30 13:10:25.897129 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:25.897081 4413 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-47mpn/perf-node-gather-daemonset-5dkw8" Oct 30 13:10:26.797769 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:26.797738 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mhfr6_7e9919e6-4abf-45ba-bcee-0237e0577a06/manager/0.log" Oct 30 13:10:29.195854 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:29.195826 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-g5g58_a63b5bb9-3521-4f1a-bfdd-08325682cfb2/server/0.log" Oct 30 13:10:30.338332 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:30.338281 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-g5g58_a63b5bb9-3521-4f1a-bfdd-08325682cfb2/server/0.log" Oct 30 13:10:30.781507 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:30.781471 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mhfr6_7e9919e6-4abf-45ba-bcee-0237e0577a06/manager/0.log" Oct 30 13:10:31.806175 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:31.806128 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mhfr6_7e9919e6-4abf-45ba-bcee-0237e0577a06/manager/0.log" Oct 30 13:10:34.305381 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:34.305346 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lqnn8_62adf1a5-f87e-4cbc-86e5-bcd2cefbd390/dns/0.log" Oct 30 13:10:34.418739 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:34.418713 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lqnn8_62adf1a5-f87e-4cbc-86e5-bcd2cefbd390/kube-rbac-proxy/0.log" Oct 30 13:10:35.037482 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:35.037458 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jz62m_535eda1f-d9e7-4908-b958-56f531b68fbb/dns-node-resolver/0.log" Oct 30 13:10:35.814061 ip-10-0-138-23 kubenswrapper[4413]: E1030 13:10:35.814025 4413 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bdxnx" containerName="extractor" Oct 30 13:10:36.868773 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:36.868740 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f64ck_27f694f1-e1de-483f-a5c5-250f52dae768/node-ca/0.log" Oct 30 13:10:37.168338 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:37.168303 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68fcc45f6c-twgp9_7eea523a-6a90-4f6b-9b89-c6c39bf6513f/kube-auth-proxy/0.log" Oct 30 13:10:37.323607 ip-10-0-138-23 kubenswrapper[4413]: I1030 13:10:37.323577 4413 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-9x4ps_b2d21fd6-1a62-4b69-af88-d5038a533f90/istio-proxy/0.log"